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Welcome  to  the  CNNA  ’92 


Cellular  Neural  Networks  (CNN)  are  locally  interconnected,  regularly  repeated,  analog 
(continuous-  or  discrete-time)  circuits  having  a  1-,  2-,  or  3-dimensional  grid  architecture.  Each 
cell  in  a  CNN  is  a  nonlinear  dynamic  system  coupled  only  to  its  nearest  neighbors.  One  ad¬ 
vantage  of  CNNs  over  competing  approaches  is  the  ease  in  implementing  VLSI  CNN  chips, 
thereby  making  real  time  operations  possible.  Although  proposed  barely  four  years  ago,  numer¬ 
ous  applications  in  the  areas  of  image  processing,  pattern  recognition,  robot  vision  and  motion 
detection  have  since  been  reported  in  several  international  journals,  conferences  and  in  the  first 
IEEE  International  Workshop  on  Cellular  Neural  Networks  and  their  Applications  in  Budapest  in 
1990.  A  special  issue  of  the  “International  Journal  on  Circuit  Theory  and  Applications”  devoted 
to  CNNs,  has  already  been  published  this  year.  Also  currently  being  prepared  and  expected  to 
appear  in  March  1 993,  is  a  special  issue  of  the  ‘Transactions  on  C'jcuits  and  Systems”. 

The  first  symposium  proved  to  be  a  truly  international  and  quite  successful  meeting.  CNNA  ’92 
has  been  able  to  attract  even  more  interesting  contributions  from  all  over  the  world.  We  ate 
anticipating  future  exciting  and  intellectually-stimulating  meetings  at  the  forthcoming  workshops 
of  this  series,  which  arc  already  being  planned  for  ’94  in  Roma,  and  ’96  in  Sevilla. 

This  Workshop  is  intended  to  share  brand-new  research  results  and  experiences  of  the  participants, 
and  to  explore  future  research  potentials.  In  support  of  these  objectives,  we  have  made  available 
a  CNN  workstation  equipped  with  a  special  hardware  accelerator  and  simulation  software  in 
order  to  provide  all  participants  the  opportunity  for  hands-on  experience.  This  will  enable 
professionals  who  arc  not  yet  active  in  this  new  and  innovative  area,  to  launch  their  own  research 
and  development  on  CNN  very  effectively. 

The  scientific  program  comprises  a  total  of  46  contributions  including  an  inaugural  lecture  by 
Professor  Chua,  who  introduced  CNNs  in  1988,  and  five  invited  presentations  by  distinguished 
experts.  The  40  papers,  which  have  been  selected  from  the  submissions,  are  forming  six  sessions 
on  Theory  (I  and  II),  Design,  Learning,  VLSI-Implementation  and  Applications.  At  the  end  of  the 
workshop  an  award  for  the  best  submitted  paper  based  on  an  evaluation  made  by  the  participants 
and  the  scientific  committee,  will  be  presented. 

We  would  like  to  express  our  sincere  thanks  to  the  members  of  the  program  committee  for  their 
effort  in  reviewing  submitted  papers  and  especially  to  all  authors  for  submitting  their  excellent 
work.  Special  thanks  are  also  extended  to  Dr.  Gloger  and  Mrs.  Theinert,  who  bore  the  main 
burden  of  making  this  workshop  organizationally  and  socially,  an  enjoyable  event  for  aU  of 
us.  Finally,  the  support  from  the  sponsoring  organizations  (IEEE,  VDE/ITG)  and  especially  the 
generous  help  from  the  European  Research  Office  of  the  US  Army  is  gratefully  acknowledged. 
We  have  the  pleasure  to  welcome  you  to  the  second  International  Workshop  on  Cellular  Neural 
Networks  and  their  Applications  (CNNA  92),  October  14-16  in  Munich,  Germany. 

We  wish  you  a  successful  conference. 


Munich,  September  1992 
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Part  1 :  The  architecture 
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Abstract 

Various  types  of  CNNs  are  summarized  and  the  taxonomy  of  CNN 
is  given  according  to  the  different  types  of  grids,  processors, 
interactions,  and  modes  of  operation.  Next,  the  CNN  Universal 
Machine  is  introduced.  The  architecture  and  the  key  features  are 
outlined.  A  quite  exhaustive  list  of  references  completes  the  paper. 


1  INTRODUCTION 

Since  the  invention  of  the  Cellular  Neural  Network  [1,2]  several  extensions  [3-8, 
etc.]  formed  the  CNN  (cellular  noi  .linear  network)  paradigm.  CNN  is  now  an 
established  field,  with  many  scientists,  engineers  and  students  contributing  to  its 
development.  The  silicon  and  optical  implementations  and  experiments  promise 
unusually  high  computing  power. 

Our  new  invention,  the  CNN  Universal  Machine  and  Supercomputer,  among 
others,  provide  three  new  capabilities:  the  analog  stored  program  and  the  time- 
multiplex  operation  of  the  layers,  the  wireless  detection,  and  flexible  representation  of 
various  PDEs  using  the  generalized  Chua’s  circuit  for  chaos  as  well. 

Next,  we  will  summarize  several  types  of  CNNs,  as  a  taxonomy.  Section  2 
contains  the  outline  of  the  basic  architecture  of  the  CNN  Universal  Machine.  In  Section 
3  some  additional  key  features  are  described.  A  fairly  exhaustive,  however  not 
complete,  list  of  references  is  attached.  The  programmability  and  key  application  areas 
are  summarized  in  a  companion  paper  in  this  volume. 
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In  the  next  table  several  types  of  CNNs  are  summarized  according  to  grid 
types,  processor  types,  interaction  types,  and  modes  of  operation. 

Most  probably,  this  table  will  ta  expanded  continuously.  In  addition,  there  are 
several  other  issues.  For  example,  the  template  design  and  learning,  the  physical 
implementations,  qualitative  theory,  accuracy,  and  the  vast  area  of  applications. 


CNN: 

Cellular  analog  programmable  multidimensional  processing  array 
with  distributed  logic  and  memory 


Grid  types 

Processor  types 

Interaction  types 

Modes  of  operation 

-  square,  hexagonal 

-  iinear  (or  smali  signal 

-  linear  (one  or  two 

-  continuous-time,  discrete-time 

tridiagonal 

operation  in  the  piece- 

variables)  memoryless 

(synchronous  or 

-  single  and  multiple 

wise-linear  sigmoid 

-  nonlinear  (one,  two,  or 

asynchronous),  time-varying 

grid-size  (coafrse 

characteristics) 

more  variables) 

-  local  mode  and  propagating 

and  fine  grid,  etc) 

-  sigmoid  (including 

memoryless 

mode 

-  equidistant  and 

unity  gain,  high 

-  delay-type 

-  fully  analog  or  combined  with 

varying  grid  size 

gain,  and  threshoiding) 

-  dynamic  jumped) 

logic  (dual  computing  CNN) 

(e.g.  logarithmic 

-  Gaussian 

-  symmetric  and 

-  fixed  template  or 

iike  in  the  retina) 

(inverse  Gaussian) 

non-symmetric 

programmable  template 

-  pianar  and  circuiar 

-  first,  second  and 

(continuously  or  in 

-  iattice  (3D) 

higher  order  (e.g.  one, 

discrete  values) 

two,  or  more  capacitors) 

-  transient-,  settling-. 

-  with  or  without  local 

oscillating-,  or  chaotic  mode 

analog  memory 

-  with  or  without  local  logic 

2 


2  The  CNN  Universe!  Machine  -  the  architecture 


First,  let  us  define  the  CNN.  CNN  is  an  analog  cellular  nonlinear  dynamic 
processor  array  characterized  by  the  following  features: 

(i)  the  analog  processors  are  processing  continuous  signals,  they  are  continuous-time 
or  discrete-time  signals  (i.e.  the  input,  output,  and  state  variables  are  C^^  functions  at 
least  for  finite  intervals),  the  processors  are  basically  identical  (a  single  term  of  them 
may  vary  regularly); 

(ii)  the  processors  are  placed  on  a  3D  geometric  cellular  grid  (several  2D  layers); 

(iii)  the  interactions  between  the  processors  are  mainly  local,  i.e.  each  processor  is 
interacting  with  others  within  a  finite  neighborhood  (nearest  neighbor  is  just  a  special 
case),  and  mainly  translation  invariant  defined  by  the  cloning  template  (the  dynamics  of 
a  given  interaction  may  be  a  design  parameter); 

(iv)  the  mode  of  operation  may  be:  transient,  equilibrium,  periodic,  chaotic,  or 
combined  with  logic  (without  A-D  conversion!). 

In  the  CNN  Universal  Machine  such  a  CNN  array  is  embedded  in  an  analogic 
(analog  and  logic)  stored  program  architecture.  This  architecture  is  shown  in  Figure  1. 

In  addition  to  the  CNN  nucleus,  each  cell  contains  several  local  analog  memory 
cells  (UVM)  and  a  single  local  logic  memory  (LLM).  LAM  cell  values  are  combined  by  a 
local  analog  output  unit  (LAOU)  while  the  logic  register  values  are  combined  by  a  local 
logic  unit  (LLU).  The  local  communication  and  control  unit  (LCCU)  provides  the  control 
of  the  cell  configuration  and  the  analog  and  logic  instruction  communication.  Some 
parts  of  this  cell  have  already  been  implemented  in  various  CNN  circuits. 

The  entire  cell  array  is  controlled  by  a  global  analogic  programming  unit 
(GAPU)  containing  an  analog  program  register  (APR),  a  logic  program  register  (LPR),  a 
switch  configuration  register  (SCR),  and  a  global  analogic  control  unit  (GACU).  This  unit 
is  a  key  for  implementing  the  analog  stored  program.  If  the  Universal  CNN  Machine  is 
implemented  in  a  single  chip,  the  CNN  universal  chip,  several  of  them  can  be  used  in 
an  array  partitioning  the  GAPU  between  the  chips  and  a  central  GAPU.  Then,  the  APR 
could  be  divided,  to  form  an  analog  cache  register. 

It  can  be  shown  that  the  local  regular  connectivity  is  essential  for  implementing 
a  stored  program  analogic  algorithm  on  silicon. 
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GAPU:  Global  analogic. 

programming  unit 


Figure  1 

3  SOME  OTHER  FEATURES 

Recently,  we  have  found  templates  [78]  which  detect  some  features  by 
switching  the  pertinent  cells  into  an  oscillation  regime.  This  allows  the  detection  by  a 
wireless  way,  thereby  drastically  increasing  the  output  throughput. 

Using  various  simple  dynamic  circuits  as  processors,  several  types  of  partial 
differential  equations  (PDEs)  can  be  modelled  in  a  discretized  space.  The  diffusion 
equation  [1 ,2]  and  the  wave  equation  [7,  64,  70]  are  such  examples.  We  can  use  the 
generalized  Chua’s  circuit  of  Figure  2  as  a  processor  (the  nonlinear  resistor  is  the  so 
called  Chua’s  diode).  Even  if  the  interactions  are  represented  by  a  simple  resistive  grid, 
and  just  a  part  of  the  circuit  is  used,  unique  phenomena  can  be  generated  [61]. 
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As  to  the  silicon  implementation  of  the  inductor,  several  equivalent  circuits  can 
be  used.  One  possibility  is  the  use  of  a  gyrator  which  can  simply  be  implemented  by 
using  two  voltage  controlled  current  sources  (or  transconductances,  used  extensively 
in  the  CNN  and  general  neural  network  implementations). 

The  CNN  Universal  Machine’s  analogic  programs  define  a  new  world  of 
analogic  (dual)  type  of  software.  All  these  functionalities  can  be  implemented  by  using 
the  CNN  Workstation  [79].  The  special  language  for  defining  these  algorithms  contains 
some  analog  instructions  as  well. 


Figu.’e  2 


Beside  the  local  connections,  without  increasing  the  connection  density,  some 
global  wires  can  also  be  used  in  the  CNN  Universal  Machine.  Local  and  global 
dynamics  are  interacting  in  this  way. 
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-  Abstract.  Based  on  two  classes  of  equi* 
librium  equations,  a  design  method  for 
reciprocal  Cellular  Neural  Networks  is 
presented.  The  local  rules  defining  the 
task  to  be  accomplished  by  the  network 
are  directly  mapped  into  a  set  of  lin¬ 
ear  inequalities  that  bound  the  solution 
space  of  the  network  parameters  for  the 
given  problem.  All  points  in  the  solu¬ 
tion  space  guarantee  the  correct  opera¬ 
tion  of  the  network.  A  solution  can  be 
computed  by  the  relaxation  method  for 
solving  sets  of  linear  inequalities. 

1  Introduction 

A  Cellular  Neural  Network  (CNN)  performs 
a  nonlinear  mapping  of  an  input  onto  an 
output  [1,  2].  The  mapping  is  completely 
defined  by  the  space-invariant  network  pa- 
r2uneters.  Each  application  such  as,  e.g., 
edge  extraction  or  shadowing  needs  its  own 
par£uneter  values  to  process  the  input  to  the 
desired  output.  Locally  connected  CNNs 
with  space-invariant  parameters  are  able  to 
process  tasks  which  can  completely  be  de¬ 
scribed  by  local  rules  involving  only  neigh¬ 
bouring  cells. 

Several  design  methods  for  synthesizing 
CNNs  have  been  proposed.  In  [3]  a  training 
rule  was  presented  to  detemoine  the  weights 
of  the  network  from  input  image  patterns 


and  desired  output  image  patterns.  By 
learning  from  examples,  it  is  assumed  that, 
after  training,  the  network  correctly  pro¬ 
cesses  inputs  which  have  never  been  shown 
before.  Unfortunately,  however,  the  pro¬ 
posed  design  method  does  not  even  guar¬ 
antee  the  desired  outputs  for  a  given  input 
learning  set. 

An  analytic  method  for  designing  simple 
CNNs  has  also  been  published  [4].  It  uses 
rules  that  explicitly  describe  the  task  to  be 
accomplished  by  the  network.  These  rules 
establish  a  set  of  inequalities  that  must  be 
satisfied  by  the  network  parameters.  A  so¬ 
lution  of  this  set  of  inequalities  guarantees 
correct  operation  of  the  network.  Neverthe¬ 
less,  it  may  be  difficult  to  construct  such  a 
set  of  inequalities  for  a  given  task,  and  for 
some  cases  the  method  is  too  restrictive,  re¬ 
sulting  in  an  empty  solution  space. 

Our  design  method  maps  the  rules  that 
define  the  application  directly  into  a  set  of 
linear  inequalities  using  two  classes  of  equi¬ 
librium  equations:  The  first  class  contains 
desired  equilibrium  states,  the  second  con¬ 
tains  forbidden  ones.  Any  point  in  the  so¬ 
lution  space  bounded  by  the  set  of  linear 
inequalities  guarantees  correct  operation  of 
the  network  for  a  given  task.  The  param¬ 
eter  values  can  be  computed  by  the  relax¬ 
ation  method  for  the  solution  of  sets  of  lin¬ 
ear  inequalities  [3]. 
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2  Desired  and  Forbidden 
Equilibrium  States 

The  first-order  nonlinear  differential  equa¬ 
tions  defining  the  dynamics  of  a  reciprocal 
CNN  with  neighbourhood  Ni(iJ)  (radius 
equ£d  to  1),  and  with  binary  input  values 
can  be  written  as  follows: 

^  +  xij  =  yja  +  ulb  +  c  (la) 
=  70* 

1  <  i  <  M  ,  1  <  i  <  N" 

where 

7o  =  [yfj  “S  l]  (1^) 


=  [  02  ‘  ‘  62  “  *  *  ^  ^  ] 


y.i(0  =  ^(l*.i(0  + 1|  -  k<i(0  - 1|)  (i<i) 

(piecewise-linear  function) 

The  CNN  is  subjected  to  the  following  re¬ 
strictions: 

|uo|  =  1  ,  V  (le) 

(binary  input) 

Oi  =  09  ,  02  =  08  . 

O3  =  O7  ,  O4  =  06  '  ^ 

(symmetry  condition) 

as  >  1  (Ig) 

(parameter  assumption) 

Table  1  illustrates  the  correspondence  be¬ 
tween  vector  a  and  the  space-invariant  clon¬ 
ing  template  A. 

The  output  values  of  the  MxN  CNN 
converge  to  ±1  for  <  — »  00  because  of  sym¬ 
metry  condition  (If)  (reciprocal  CNN),  pa¬ 
rameter  assumption  (Ig)  and  the  sigmoidal 
characteristic  (Id)  of  the  nonlinear  function 
[1].  This  means,  that  each  vector  compo¬ 
nent  Y/,  1  <  r  <  9,  converges  to  ±1  for 


ai 

02 

as 

04 

as 

06 

07 

a% 

09 

Table  1:  Component  correspondence  for 
vector  a  and  template  A 

t  — »  00.  On  the  other  hand,  because  of  the 
binary-input  condition  (le),  I'y’-’j  =  1,  10  < 
r  <  18,  for  t  >  0.  Thus,  7y  (t  00)  = 
7^  €  {—1,  1}^®  contains  binary  compo¬ 
nents  ±1  corresponding  to  output  and  in¬ 
put  values  in  the  neighbourhood  Ni{i,j)  of 
cell  C(i,i).  * 

Let  r  be  the  set  of  all  possible  vectors  7“ 
with  different  component  values.  Next  let 
Fj  and  F/  be  disjoint  subsets  of  F  contain¬ 
ing  those  desired  and  forbidden  vectors  ‘^7“ 
and  ^7“,  whose  components  correspond  to 
input /output  combinations  which  are  pre¬ 
scribed  by,  resp.  should  not  appear  in,  a 
specific  application,  i.e.  a  specific  mapping 
function  F 

F  :  {-1,  1}*^^^  — .  {-1,  (2) 

Then  for  all  7~  €  F^,  i.e.  for  vectors  **7”, 
the  following  inequalities  formulate  the  de¬ 
sired  conditions  that  the  parameter  vector 
X  must  satisfy  in  order  to  enable  the  CNN 
to  settle  in  a  desired  output  state: 

7?'*  i  i,  tor  7?  =  1  (3a) 
7^*  <  -1,  for  7?  =  -1  (3b) 

For  clarity,  the  shorter  notation  7]^  has  been 
used  instead  of  (**7^)^.  * 

On  the  other  hand,  vectors  ^7“  belong¬ 
ing  to  the  set  F/  of  forbidden  combinations 

‘The  last  component  of  vector  7"  is  always 
equal  to  unity  due  to  7!'^  :=  1  (see  (lb)),  and  is 
not  related  to  input  or  output  values. 

’Inequalities  (3)  result  from  equations  (la)  and 
(Id),  and  from  the  facts  that  =  0  and  ]Hj  =  ±1 
for  I  -*  00  (1). 
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define  the  following  inequalities  that  pre¬ 
vent  the  transient  of  the  CNN  to  converge 
to  an  undesired  output  state  for  a  given  in¬ 
put: 

75*  <  1,  for  'Ti  =  1  (4a) 

7J*  >  -1,  for  'Ts  =  -1  (4b) 

Here  7^  stands  for  (■^7“)^.  With  inequal¬ 
ities  (4)  and  for  a  fixed  parameter  vector 
X,  any  state  variable  Xij  that  produces  an 
undesired  output  j/,j  =  ±1  is  pushed  back 
from  the  outer  parts  of  the  piecewise-linear 
function  into  the  linear  region  |z,j|  <  1  in 
order  to  let  the  CNN  search  for  another 
equilibrium  state.  If  the  resulting  vector 
7,j  of  this  new  equilibrium  state  also  be¬ 
longs  to  F/  then  the  state  variable  is  pushed 
back  again  until  a  desired  equilibrium  state 
is  reached.  Inequalities  (3)  assure  that  such 
an  equilibrium  state  exists. 

In  summary,  inequalities  (3)  and  (4)  to¬ 
gether  with  (Ig)  and  symmetry  condition 
(If)  bound  the  solution  space  of  parame¬ 
ter  vector  *.  The  correct  operation  of  the 
CNN  for  any  point  within  the  solution  space 
is  guaranteed  by  this  procedure.  The  ele¬ 
ments  of  Frf  and  F /  can  directly  be  d^;rived 
from  the  local  rules  defining  F,  as  demon¬ 
strated  by  the  next  example. 

3  Example: 

Edge  Extraction 

The  task  of  extracting  the  edge  of  a  figure 
can  be  completely  described  by  local  rules 
which  only  involve  cells  C{k,  1)  belonging  to 
the  corresponding  neighbourhood  Ni(i,j). 
This  is  essential  because  applications  which 
need  the  direct  influence  of  a  cell  C(k,l)  ^ 
Ni{i,j)  on  C{i,j)  cannot  be  realized  with 
the  common  definition  (1)  of  a  CNN.  The 
advantage  of  using  local  rules  for  the  de¬ 
scription  of  the  task  is  that  the  correct  op¬ 
eration  of  the  network  does  not  depend  on 


its  size,  i.e.  on  the  values  of  constants  M 
and  N.  The  rules  for  edge  extraction  can  be 
stated  as  follows  (compare  with  Figure  1): 

A  cell  does  not  belong  to  the  edge  of  a 
figure  if 

•  the  cell  does  not  belong  to  the  figure 
itself,  i.e.  if  its  input  value  is  white 
(or,  equivalently,  —1) 

•  the  cell  is  in  the  inner  part  of  the  fig¬ 

ure,  i.e.  if  its  input  value  is  black  (or 
equivalently  1)  and  the  input  values  of 
cells C(i-1,;),  C(i,j  +  1) 

and  (7(1  -I- 1,  j)  are  also  black. 

These  rules  deal  only  with  five  cells,  namely 
the  center  cell  and  the  off-diagonal  cells 
within  the  neighbourhood.  For  the  center 
cell  the  input  and  the  output  values  are  re¬ 
quired,  for  the  off-diagonal  cells  only  the  in¬ 
put  value  is  needed  (see  Table  2).  Moreover, 


Table  2:  inputs  and  the  output  involved  in 
the  edge-extraction  problem 

the  space  symmetry  of  the  edge-extraction 
problem  allows  us  to  weight  the  off-diago¬ 
nal  input  values  with  the  same  parameter 
6#.  Thus,  a  reduced  parameter  vector  is 
sufficient  for  the  correct  operation  of  the 
network: 

=  [  Cs  6#  65  c  ] ,  (5a) 

where 

5^  =  62  =  64  =  65  =  is 

Vectors  7,j  and  7“  are  modified  and  now 
contain  only  the  needed  input  and  output 
values  having  again  the  same  size  as  param¬ 
eter  vector  x: 

tJ-  =  [75'  7n  +  713  +  7x5  +  717  Yu  1  ] 

=>  7.7  =  hi'  -7?  ‘7?  1  ]'’  (5b) 
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Figure  1:  (a)  input  pattern  and 

(b)  the  desired  output  pattern  for  edge  extraction  in  a  16x16  layer  ^ 


Vector  component  corresponds  to  the 
output  of  cell  C(t,  j),  *73  contains  its  input 
value,  and  *73^  is  the  sum  of  the  off-diagonal 
inputs  around  cell  C{i,j).  *73^  and  *7^^  can 
take  on  values  ±1,  i.e.  the  cell’s  input  and 
output  are  either  black  or  white. 

Next,  we  have  to  differentiate  between 
cells  at  a  comer,  at  the  border,  or  in  the 
inner  part  of  the  CNN  in  order  to  compute 
the  values  of  vector  component  *73'.  It  is 
obvious  that  for  an  inner  cell  *^2 
on  values  —4,  —2,  0,  2,  4  depending  on  the 
sum  of  white  (=  —1)  and  black  (=  1)  off- 
diagonal  cells  ^  around  cell  For  ex¬ 

ample,  if  C{i,j)  is  surroimded  by  white  off- 
diagonal  cells  then  *73’  =  — 4,  but  if  one  of 
these  cells  is  black  then  *73^  =  -3-1-1  =  —2. 
For  border  cells,  one  of  the  elements  of  the 
sum  7Ji  +713+716 +717  is  zero  because  there 
cell  C(i,j)  is  surrounded  only  by  three  off- 
diagonal  cells.  For  comer  cells  even  two  ele¬ 
ments  of  this  sum  disappear.  The  following 
relations  summarize  the  results: 

*7?,  *7?'  €  {-1,  1  }  (6a) 


*We  take  the  same  two-dimensional  image  as 
was  used  in  [3]  to  train  the  network. 

^The  input  values  of  the  cells  are  added. 


•-r?  6  {-2,  0,  2  } 

(foi  comer  cells) 

•7?  e  {-3,-1, 1,  3} 

(for  border  cells) 

•7?  €  {-4,-2,  0,  2,  4} 

(for  center  cells) 

From  the  first  rule  of  the  edge-extraction 
problem  it  follows  that  if  the  input  of  a  cell 
is  white  then  the  cell  has  to  produce  a  white 
output,  i.e.  whenever  •73'  =  —  1  the  out¬ 
put  •7j'-  must  also  be  —1.  The  second  rule 
states  that  whenever  *73^  =  1  (black  input), 
*71^  also  has  to  be  1  unless  the  inputs  of  the 
off-diagonal  cells  are  all  black,  i.e.  *73'  =  4. 
FVom  this,  the  set  can  be  constmcted  in 
a  straight-forward  procedure  assigning  to  it 
all  vectors  yff  whose  components  accom¬ 
plish  one  of  the  following  conditions: 

(0  *7?  =  -1  A  *7?  =  -1 

(«)  *73^  =  1  A  •7J^  =  1  A 

*7?  €{-4,-3, -2,-1,  0,  1,  2,  3} 

(**»)  *73  =  1  A  *71^  =  -1  A  *72^  =  4 

Equation  (7)  shows  the  result. 
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r /  can  easily  be  constructed  using  the 
disjoint  property  of  both  sets  Fj  and  F /  snd 
the  fact  that  all  other  vectors  7“  whose 
components  do  not  accomplish  any  of  the 
conditions  (i)  -  (tit)  lead  to  forbidden  equi¬ 
librium  states: 


V,  =  r\ri  (8) 

We  now  have  all  the  inequalities  that 
bound  the  solution  space  of  parameter  vec¬ 
tor  X  €  such  that  any  point  in  the  solu¬ 
tion  space  guarantees  the  correct  operation 
of  the  CNN  for  edge  extraction,  indepen¬ 
dently  of  the  sizes  M  and  N  and  the  initial 
state^.  A  solution  that  satisfies  inequali¬ 
ties  (Ig),  (3)  and  (4)  can  be  computed  by 
the  relaxation  method  for  sol'/ing  sets  of  lin¬ 
ear  inequalities  [3].  The  following  equation 
gives  a  solution  of  the  parameter  vector  x 
for  the  edge-extraction  problem: 


■  1.0559  ■ 

-0.2615 
®  ”  1.2436 

.  -0.3419  . 


(9) 


Table  3  orders  the  same  result  in  template 
notation. 

‘‘The  local  rules  for  edge  extraction  do  not  in¬ 
volve  the  initial  state  of  the  CNN. 


cloning  template  B 


cloning  template  A 

I  =  -0.3419 
constant  current  source 

Table  3:  Eldge  Extraction:  same  parameter 
set  as  in  equation  (9)  ordered  in 
template  notation 

4  Conclusions 

An  exact  design  method  for  reciprocal  Cel¬ 
lular  Neural  Networks  with  binary  inputs 
has  been  presented.  Given  an  application, 
two  disjoint  sets  of  vectors  can  be  derived 
directly  from  the  local  rules  that  define  the 
task  to  be  accomplished  by  the  network. 
The  sets  contain  desired  and  forbidden  in¬ 
put/output  combinations,  resp.,  which  lead 
to  appropriate  inequalities  bounding  the  pa¬ 
rameter  space.  Any  point  in  the  solution 
space,  i.e.  any  parameter  set  that  satisfies 
all  inequalities,  guarwtees  the  correct  op- 


eration  of  the  CNN  for  the  given  task. 

The  exact  design  method  has  been  demon¬ 
strated  in  detail  for  the  problem  of  edge  ex¬ 
traction.  A  CNN  with  a  parameter  set  that 
satisfies  the  constructed  inequalities  extracts 
the  edge  of  any  pattern  at  any  position  in  a 
two-dimensional  layer  of  arbitrary  size,  in¬ 
dependently  of  the  initial  state. 
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Abstract 

In  this  paper  are  indtoduced  b  isic  design  methods  for  the  class  of  cellular  neural 
networks  (CNN’ s)  with  continuous  input  signals.  The  realistic  Modell  of  CNN’ s 
are  proposed  by  Chua  and  Yang  [2,  3]  and  combine  components  of  the  Hopfield-Net 
[8],  the  Cellular  Automatas  [13]  und  der  Cellular  Systems  [14].  The  CNN  design 
methods  integrate  special  conditions  for  technical  architectures  with  respect  to  real¬ 
time  implementations.  Haeijan’s  polynomial  solution  method  [5]  are  applied  to  solve 
the  set  of  linear  inequalities  which  correspond  with  the  CNN  design. 


1  Introduction 


A  CNN  architecture  for  image  processing  are  marked  by  regular  arrangement  of  identical 
cells,  which  are  shown  as  rectangular  5x8  CNN  with  40  cells  c*'  for  1  <  A:  <  40  left  in 
figure  1.  All  cells  characterize  an  identical  implementation,  that  is  chosen  as  a  discret-time 
system  [1,  4,  6,  11]  and  follows  as  signal  flow  graph  right  in  figure  1. 


Figure  1;  Arrangement  of  cells  and  signal  flow  graph 

All  cells  performe  weighted  sums  of  local  in-  and  output  signals  which  delayed  by  a 
discrete-time  and  pass  the  results  through  nonlinearities  into  the  new  output  signals. 
Figure  2  shows  all  vector  elementes  of  input,  output  and  weight  connections,  which  are 
identical  constructed  around  all  cells  c*’’. 


?//4l(U 

bj62(0 

^*43(0 

J^44(U 

P//2(0 

bjjait) 

“/t2(Ul«/»3(0 

0/44(0 

Figure  2:  Arrangement  of  vector  elementes 
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The  following  table  1  presents  the  denominations  of  all  quantities. 


IM  =  {— 1,0, +1}  :=  {white  ,  grey,  black), 

P  £  IN  feedforward  connections, 
y^{t)  €  IM  :  output  signal  of  c*', 

Ujjit)  €  [—1,  +1]^  :  feedforward  vector  of  c*", 
G  IR^  :  feedback  weights, 

I{t)  G  IR  :  Source  and 


0  E  IN  :  feedback  connections, 

G  [— 1,+1]  :  input  signal  of  c*, 
y/bCO  ^  ;  feedback  vector  of  c^, 

:=  (y)^(t),u)J(0,l)  e  iR^^^^\ 

b//(<)  G  Ilt^  :  feedforward  weights, 

wT(t)  :=  (aJ,(t),bJ^(0,/W)  e  IRP^f^+\ 


Table  1:  The  denominations  of  all  quantities 


The  positions  of  the  vector  elements  for  the  in-,  outputs  and  the  constant  1  show  figure 
2  and  table  1.  The  same  form  for  the  temporal  variable  weights  or  “Cloning  Template” 
of  the  in-,  outputs  and  source  follows  also  in  figure  2  and  table  1,  which  is  identical  for 
all  cells.  The  equations  (1,  2,  3  and  4)  represent  the  mathematical  description  of  the 
dynamical  behaviour  of  ail  cells  in  all  discrete-times  t  with  t  G  {0,  . .  .},tA  G  IR+- 

The  sums  of  the  weighted  in-  and  outputs  and  source 

s\t)  =  ajft(t)y}i(<)  +  h]j{t)u'}j{t)  +  I{t)  (1) 

are  transformed  with  the  vectors  of  in-  and  ou'^..UoS  v*(t)  and  weights  w(t)  in  the  sums 

,‘(i)  =  W^(()v‘(i).  (2) 


The  delayed  step  of  the  sums  are  loted  down  with  the  states  x^(t)  in  equation  (3), 


x‘‘(t)  =  s'‘{i  —  ^a) 


(3) 


The  nonlinearities  perform  the  states  into  the  outputs  and  are  shown  in  figure  3. 

-1  V  -3<x‘(f)<-l 

0  V  -l<x^{t)<+l  (4) 

1  V  -f-1  <  x*(f)  <  +3. 


0 


1 

Figure  3;  Stepped  nonlinearity 

The  iteration  equation  (5)  arise  out  of  the  equations(3  and  4),  which  calculates  the  present 
outputs  with  the  past  in-  and  outputs  and  cloning  templats. 

f  -1  V  -3  <  s''(t  -<a)  <  -1 

/(/)  =  {  0  V  -1  <s*(<-fA)  < +1  (5) 

[  "f  1  V  -|-1  <  s^[t  —  ^a)  <  "HS. 

The  bounds  of  the  outputs  [9]  follow  with  table  1  und  equation  (4) 

V  =►  y\t)ElM.  (6) 

The  CNN’ s  perform  out  of  all  initial  values  u*(0),y*(0),  w(0)  with  all  sequences  of  oper¬ 
ations  w{f)  and  inputs  «*'(<)  a  sequence  of  outputs  y*(<). 
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2  Design 

The  application  of  CNN’ s  for  image  processing  are  determined  by  the  CNN  design  meth¬ 
ods  for  the  determination  [7,  10,  11,  12,  15]  of  the  temporal  sequence  of  local  operations 
w(t).  All  values  of  weights  sums  and  states  are  restricted  into  intervals. 

K(t)|<2,  |s'=(0l  =  1x^01  <  3.  (7) 

The  start  points  of  the  design  form  the  explicite  iteration  equation  (5)  and  equation  (2) 
which  characterize  the  relation  between  the  identical  local  connected  inputs  u^(t),  outputs 
and  the  cloning  templates  w(t). 

For  a  given  temporal  sequence  of  in-  and  outputs  inspects  the  following  design  method 
the  existence  of  the  sequence  of  cloning  templates  which  perform  the  given  sequence  of 
outputs.  For  this  purpose  are  estimated  in  equation  (5)  the  linear  sums  —  look  at 
the  design  method  (8),  between  their  extreme  points  with  respect  to  the  following  output 
j/*(Z)  by  linear  inequalities  for  all  cells. 

i f  y^{t)  =  — 1  — 3  <  s'‘(t  —  Za)  <  —1 

then 

v*'''’(Z  -  Za)w(Z  -  Za)  >  —3  v^'^(Z  -  Za)w(Z  -  Za)  <  -1 
if  y^{t)  =  0  =>  — 1  <  s*'(Z  —  Za)  <  +1 

then  (8) 

v^''’(Z  -  Za)w(Z  -  Za)  >  —1  v^^{t  -  Za)w(Z  -  Za)  <  Tl 

if  =  +1  =>  +1  <  5*-'(Z  -  Za)  <  +3 

then 

v*^(Z  -  Za)w(/  -  Za)  >  -1-1  -  Za)w(Z  -  Za)  <  -!-3 

The  further  technique  are  introduced  by  a  visual  example.  The  detection  of  black-white 
differences  in  images  can  be  solved  by  a  feedforward  and  a  feedback  connection  v*^(Z)  = 
(u*’(Z),  y^(Z)).  The  two  images  are  assigned  to  the  inputs  u*(Z  —  Za)  and  outputs  7/^{Z  —  Za) 
and  the  differences  are  assigned  to  the  next  outputs  J/*(Z).  Figure  4  shows  all  transients 
and  Table  2  shows  their  values. 

□  Um-m  BD-ED  BB-B 


Figure  4;  Transients 


■ 

1 

(-1,-1) 

-1 

IB 

(+1,-1) 

0 

2 

■aiBMM 

0 

B 

(+1,4-1) 

+1 

Table  2;  In-  und  output  values  of  differences 


The  design  methods  (8)  and  restrictions  (7)  lead  to  the  inequalities  (9)  with  the  set  of 
solutions  in  figure  8. 

-fioi  -f  tz;2  >  -i-1 

-i-wi  -  twj  >  -1  (9) 

-Wi  -  U’2  >  -f  1 
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-  ^  0  1^(0 


Figure  5 

The  intervals  of  input,  output  and  sum  values  v^,  and  sa,  see  figure  5  ,  are  shown  in 
equation  (10)  for  =  0. 

*7  =  =>  -1  +  SA  <  -  ^a)  < +1  - -sa 

then  (10) 

(v*''’(t  -  ^a)  ±  VA)w(t  -  ^a)  >  -1  4-  SA,  (v^''’(t  -  ^a)  ±  VA)w(t  -  /a)  <  +1  -  3a 

All  combinations  of  values,  see  figure  6,  lead  to  a  continuous  set  of  linear  inequalities  in 
figure  7.  This  set  of  solutions  can  be  estimated  by  the  finit  number  of  extreme  inequalities. 
The  number  is  and  the  combinations  follows  with  a  binary  system  from  0  to 


Figure  6  Figure  7 

Figure  9  shows  the  solutions  of  (9)  which  are  estimated  by  the  inequalities  of  figure  7. 


Figure  8  Figure  9 

A  stabil  technical  implementation  [10]  of  realistic  cloning  template  leads  to  a  centred 

point  of  the  maximal  ball  with  radius  r  within  the  set  of  solutions  in  figure  9.  This 
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cloning  template  is  stabil  for  all  defects  of  the  cloning  template  which  are  lower  than  the 
maximal  radius. 

v^w  +  C+ I  V  1  r  >  0,  r  e  IR  (11) 

For  the  determination  of  the  maximal  ball  and  radius  are  introduced  parameter  [10  and 
11]  in  all  inequalities  which  are  present  in  equation  (11,  12). 

v'''^(t-tA)w(t-tA)  +  C^+ I  1  r  >  0 

(UieiR  (>2) 

All  inequalities  of  discrete  step  for  a  parameter  build  a  set  of  inequalities  (13),  which  can 
be  solved  by  Hacijan’s  polynomial  solution  method  [5]. 

<  0^  (13) 


The  polynomial  solution  method  constructs  a  sequence  a  of  ellipses  which  approximate 
the  set  of  solutions  and  present  a  solution  by  the  characteristic  central  point  x‘' 
tranformation  matrix  0*^.  The  method  are  denoted  in  the  following 


step  0:  <7  =  =  0,  0''  =  (0  +  P  +  1)22E 

step  1:  if  Vtt  e  {1,. . .  ,n}ajx‘'  < 
then  stop  with  solution 
else  qJx*"  >  0ir 


O.Ix‘’-0n 


6^  =  0^  = 
step  2:  =  x"  -  - 


(A+1){1-/3„)52J 


a  =  a  +  1  goto  step  1 

7r,n,a,A  €  IN]  €  IR]  ar,X^‘*2r  €  IR\  €  IR^^\ 


If  a  solution  exist  for  every  discrete  step  t  in  ?.ll  sets  of  inequa-lities  (12  and  13)  then  all 
solutions  build  the  sequence  of  cloning  templats  w(t). 
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Abstract 

This  paper  describes  general  methods  of  designing  a  discrete-time  cellular 
neural  network  implementing  an  arbitrary  Boolean  function  defined  on  the 
r -neighbourhood.  This  is  achieved  by  operating  the  network  with 
time-variant  templates  as  a  cellular  automaton  that  processes  only  binary 
inputs.  These  methods  are  suital  le  for  solving  local  tasks.  As  an  example, 
testing  minimal  distances  is  discussed. 


1.  INTRODUCTION 

Discrete-time  cellular  neural  network  as  introduced  in  [3]  is  a  large  scale 
nonlinear  circuit,  composed  of  locally  connected  cells.  It  has  a  simple 
nonlinear  circuit  at  each  node  and  is  very  well  suited  for  VLSI 
implementations. 

In  this  paper  we  restrict  ourselves  to  binary  valued  operations  and  study 
how  to  implement  local  Boolean  functions  using  DTCNNs.  We  prove  that  every 
Boolean  function  defined  on  the  r-neighbourhood  can  be  implemented  in  a 
single  layer  DTCNN  with  time-variant  templates.  This  is  achieved  by 
operating  the  network  as  a  cellular  automaton  that  processes  only  binary 
inputs.  General  methods  for  designing  DTCNNs  Implementing  Boolean  functions 
are  presented.  They  are  based  on  a  digital  convolution.  In  every  iteration 
step  the  output  state  of  any  cell  can  be  changed  only  if  the  input  values 
within  the  r-neighbourhood  match  exactly  the  given  pattern,  stored  in  the 
control  operator.  The  coefficients  of  control  operator  are  restricted  to 
the  set  {-1,0, 1>,  while  the  feedback  operator  consists  of  self-feedback 
only. 

Presented  methods  are  suitable  ior  solving  local  tasks.  In  the  second  part 
of  the  paper  a  new  application  of  DTCNN  is  described.  Testing  distances 
between  objects  for  an  arbitrary  value  of  the  minimal  distance  is  performed 
by  a  two-layer  network  with  time-variant  templates. 


2.  NOTATIONS 


Definition  1:  Discrete-time  cellular  neural  network  (DTCNN)  using 
time-variant  templates  is  defined  by  the  following  recursive  algorithm: 


for  k>0 


x‘=(k)  =  a®(k)-y‘*(k-l) 

d 

y®(k)  -  f(x*(k))  =  I 


+  b®(k)*u‘*  +  i‘^(k) 

d 

1  for  x‘^(k)fc0 
-1  for  x'^(k)<0 


(1) 

(2) 


0-7803-0875-1/92  $3.00.©1992IEEE 
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In  (1)  the  Einstein  summation  convention  [7]  is  used.  For  its  convenience 
it  will  be  used  throughout  the  paper. 

The  feedback  coefficients  a^(k),  the  control  coefficients  b'^Ck)  and  the 

d  d 

thresholds  i'^Ck)  are  real  parameters.  The  initial  state  y*^(0),  the  output 

state  after  the  k^**  iteration  y'^(k)  and  the  input  u*^  of  a  cell  c  are  binary 
valued. 

The  cells  of  the  network  form  a  regular  grid.  The  cells  are  only  locally 
connected.  By  the  definition  the  template  coefficients  are  translationally 
invariant.  A  single  template  for  given  k  is  called  a  snapshot  template. 

This  definition  differs  from  the  one  given  in  [2]  in  that  the  input  values 
u  are  assumed  to  be  constant,  while  originally  they  have  been 
time-variant. 

Let  r  be  a  constant  integer  value.  Let  us  consider  a  cell  c.  The 
r-neighbourhood  of  the  cell  c  is  defined  as  the  set  of  all  cells  within  the 
distance  r,  with  respect  to  the  ’maximum’  metric.  It  is  denoted  by  N  (c). 

Let  M=M(r)  be  the  cardinality  of  N^(c).  For  each  cell  its  r-neighbourhood 
contains  M(r)=(2r+1)^  cells,  when  restricting  to  square  grids. 

Let  d  ,d . d  denote  the  cells  belonging  to  N  (c).  U  :  =(u^ ,  .  .  .  , u**)  is 

1  2  M  r  c 

the  vector  of  the  inputs  of  all  cells  within  the  r-neighbourhood  of  the 
cell  c. 

Let  S  be  the  set  of  all  possible  binary  input  patterns  within  N  (c): 

r 

S  =  <u=(u\...u’‘)6{-l,+l}‘'}  (3) 

Definition  2;  Let  g  be  an  arbitrary  Boolean  function  defined  on  S: 

g:  S  — »  {-1,+1> 

where  +1  is  interpreted  as  true  and  -1  as  false. 

We  say  that  a  DTCNN  implements  function  g  if  there  exists  k^-0  such  that 

Vc  VU  €S  Vkak  y‘'(k)=g(U  ) 

c  0  c 


3.  DESIGNING  OF  DTCNN  PERFORMING  BOOLEAN  FUNCTIONS 

Two  methods  of  designing  DTCNNs  implementing  an  arbitrary  Boolean  function 
can  be  proposed. 

Method  1:  In  order  to  implement  an  arbitrary  Boolean  function  perform  the 
following  two  steps: 

Step  1:  Let  us  write  the  Boolean  function  g  in  a  sum-of -products  form: 

g(u^ . U**)  =  (u^’^A.  .  .  )v.  .  .  vCu^’^A.  .  .  Au"’"*"*  )  (4) 

We  have  n  products.  The  k*'*’  product  has  w(k)  factors. 

u*'’^  denotes  one  of  the  variables  u^,...u**,  which  may  be  negated. 

Step  2:  Let  n  be  the  number  of  snapshot  templates. 

Vc  Vk€{l, . . . ,n} 


y^(0)=-l,  i‘'(k)=+l. 

a'^(k)=w(k),  a‘^(k)=0  for  c*d, 
c  cl 

for  a=d  eN  (c)={d  .d . d  }  : 

s  r  12  M 

b^(k)=+l  if  is  present  in  the  k  product  of  (4) 

b^(k)=-l  if  is  present  in  the  k*’*'  product 

b^(k)=0  if  u^  and  are  not  present  in  the  k*’*'  product 
for  d(i5N  (c)  ;  b''(k)=0. 

r  cl 

The  binary  valued  input  pattern  is  the  information  to  be  processed,  n 
iterations  with  tirae-variant  templates  should  be  perforraed.  Then  we  can 
either  stop  processing  or  we  can  continue  with  the  last  snapshot  template. 

Proof  of  the  correctness  of  method  1: 

First  let  us  consider  an  arbitrary  cell  c.  any  integer  k6{l,...,ri>  and 
several  cases  depending  on  the  value  of  y  (k-l)  and  the  input  vector  u. 

Case  1 :  y'^(k-l)=+l 

a'^Ckj-y^'Ck-l)  +  i^^Ck)  =  w  +1 

d  ■'  k 

Because  exactly  w^  elements  of  b^(k)  (for  given  c)  are  equal  to  +1  or  -1 
and  other  elements  are  0  then; 

lb''(k)-u'*|  s  Ib'^Ck)  j  -1“*  =  w 

d  •  *  <j  '  It 

Hence  x'(k)>0  and  y‘^(k)=+l. 

Case  ?.:  y'(k-l)=-l  and  Vd  :  b°(k)=u‘‘  or  b‘'(k)=0 

d  d 

a^(k)  •  y'^Ck-l )  i^^Ck)  =  +  1 

b^CkJ-u**  =  lb‘^(k)i-l‘*  =  w 
d  '  d  k 

Hence  y'^(k)=f(-w  +l+w  )  =  +1. 

k  k 

Case  3:  y°(k-l)=-l  and  3c.  ;  (k)*0 

d 

b'(k)-u‘*  ^  !b^(k)  I  •  1**  -  2  =  w  -  2 

d  '  d  '  k 

x'^(k)  =  a‘^(k)-y'*(k-l)  +  i'^Ck)  +  b‘^(k)-u'*  s-w  +  l+  w  -  2  =  -l 

d  ^  d  k  k 

In  this  case  y°(k)=-l. 

It  effects  from  cases  1-3  that  a  transition  from  +1  to  -1  is  not  allowed 
and  a  transition  from  -1  to  +1  occurs  in  step  k  only  if  the  input  pattern 
in  the  r-neighbourhood  exactly  matches  the  non-zero  values  of  the  matrix 
b(k).  The  function  g  is  written  in  the  sum-of -products  form  and  every 
iteration  step  computes  one  product  of  this  sum. 

As  y‘^(0)=-l  then  y‘^{n)=+i  if  and  only  if  3k€{l,...,n}  such  that 

Vd  :  b*^(k)=u‘*  or  b‘^(k)=0. 

d  d 

Because  for  every  kan  y*^(k)=y  (n),  the  function  g  is  implemented  properly. 


Method  2:  Let  us  write  function  g  in  a  product-of-surns  form: 

/I  K.  ,  1,1  1,h(1).  f  n,l  n,w(n) ^ 

g(u  ,  .  .  .  ,U  )  =  (U  V.  .  .VU  *  )a.  .  . A (u  V.  .  . VU  ) 
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All  parameters  of  the  network  should  be  the  same  as  in  method  1  with  the 
following  differences:  i‘^(k)=-l  (instead  of  +1),  y'^(0)=+l  (instead  of  -1). 

Proof  for  method  2  is  dual  to  that  of  method  1. 


Remark:  Both  methods  can  be  used  for  the  implementation  of  an  arbitrary 
Boolean  function.  This  can  be  seen  by  noting  that  any  Boolean  function  can 
be  written  in  both  the  sum-of -products  and  the  product-of-sums  forms  [4]. 
The  methods  1  and  2  usually  solve  the  same  problem  with  the  different 
complexity. 

Example:  We  want  to  detect  horizontal  and  vertical  black  lines  of  length 
greater  than  two  pixels.  This  is  equivalent  to  detecting  all  black  pixels 
which  have  their  horizontally  adjacent  pixels  black  or  which  have  their 
vertically  adjacent  pixels  black. 


This  function  can  be  presented  in  the  sum-of-products  form  as: 

,1  9.  ,  2  5  8.  .  4  5  6. 

g(u . U  )  =  (u  AU  AU  )v(u  AU  AU  ) 

Method  1  leeds  to  the  following  solution:  n=2,  Vc  y‘^(0)=-l,  u=image. 


0  1  0 

0  0  0 

a ( 1 ) =a ( 2 ) = 

4 

.  b(l)= 

0  1  0 

0  10 

II 

w 

1  1  1 

0  0  0 

4.  TESTING  MINIMAL  DISTANCES 

The  task  is  to  detect  black  objects,  which  are  closer  then  a  distance  of  k 
pixels,  where  k  is  a  given  constant  integer  value.  The  solution  of  this 
problem  can  be  used  in  layout  design  for  checking  if  any  two  objects  are 
too  close  to  each  other. 

The  solution  for  k=2  described  in  [5]  is  based  on  extracting  black  pixels 
having  a  white  neighbour  in  a  given  direction  and  a  black  second  neighbour 
in  the  same  direction.  It  can  be  done  with  2-neighbourhood  templates.  This 
solution  allows  to  extract  pixels  in  one  direction.  To  be  able  to  solve 
this  problem  for  four  directions  it  was  proposed  to  process  simultaneously 
but  separately  in  four  direction  and  then  perform  the  OR  function  on  the 
outputs. 

In  this  paper  we  use  DTCNN  architecture  with  time-variant  templates  for 
solving  the  problem  for  an  arbitrary  value  of  minimal  distance.  Each 
snapshot  template  is  equivalent  to  one  layer  of  hardware  [5]  while  the  OR 
function  is  performed  at  the  same  time  by  choosing  templates  properly. 

Before  the  solution  of  that  problem  is  given  some  definitions  have  to  be 
introduced. 

Let  the  distance  between  two  pixels  w, z  be  defined  by  the  ’maximum’  metric: 

d(w,z):=max{lw^-zj,  Iw^-z^l) 
where  are  the  Integer  coordinates  of  the  pixels. 

Definition  3:  Two  black  pixels  w  and  z  are  called  k-separated,  if  their 
distance  is  k  and  they  cannot  be  connected  by  a  black  four-connected  path 
completely  enclosed  in  the  rectangle  determined  by  w  and  z,  where: 

o  the  four -connected  path  is  a  sequence  of  orthogonally  adjacent  pixels 
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[such  pixels  are  also  called  four -connected  pixels  (2)), 

o  the  rectangle  determined  by  w  and  z  is  the  minimal  rectangle  containing 
w  and  z  with  horizontal  and  vertical  sides. 


Detection  of  1-separated  pixels 

The  task  is  to  find  all  1-separated  pixels  and  mark  their  positions  on  the 
grid.  The  solution  is  based  on  searching  for  white  pixels  which  have 
1-separated  black  neighbours.  This  problem  can  be  represented  in  the  form 
of  a  Boolean  function  as: 


1  9,  ,  S  6  a  9,  ,  S  2  6  3.  ,  5  2  4  1  ,  S  4  8  7, 

f  (u  ,  .  .  .  U  )  =  (u  AU  AU  AU  )v(u  AU  AU  AU  )V(u  AU  AU  AU  )v(u  AU  AU  AU  J 


With  method  1,  we  can  construct  the  following  DTCNN: 
n=4,  y(0)=-l.  u=image,  a(k)=4,  i(k)=l  for  every  ke{1..4> 


- 1 

0  0  0 

0  1  -1 

-1  1  0 

0  0  0 

b(l)  = 

0  -1  1 

0  1  -1 

,  b(2)= 

0  -1  1 

0  0  0 

,  b(3)= 

1  -1  0 

0  0  0 

,  b(4)= 

1  -1  0 

-1  1  0 

In  this  solution  each  detected  pair  of  1-separated  pixels  is  marked  by  two 
black  pixels  in  the  output.  We  can  modify  the  network  in  the  following  way: 
n=2,  the  templates  for  k=l,2  are  the  same  and  stii i  all  1-separated  pixels 
are  detected. 

Similar  networks  can  be  used  for  detection  of  2  and  3-separated  pixels. 
When  searching  for  2-separated  pixels  we  need  8  snapshot  templates  and 
1 -neighbourhood.  For  detection  of  3-separated  pixels  we  need  12  snapshot 
templates  and  2-neighbourhood  of  matrix  b. 

Detection  of  k-separated  pixels 

We  show  how  to  use  the  solution  for  k=2,3  to  build  a  2-layer  DTCNN  which 
detects  k-separated  pixels  for  every  k.  The  idea  is  as  fellows: 

j:  =2; 
repeat 

detect  2-scparated  pixels; {equivalent  to  detection  of  j-separated  pixels} 
detect  3-separated  pixels; {equiv,  to  detection  of  ( j+1 )-separated  pixels} 
increase  objects  by  one  pixel  in  every  direction; 
until  (j>k); 

Increasing  objects  rely  on  converting  white  pixels  to  black  if  any  of  their 
eight-connected  neighbours  is  black.  The  DTCNN  performing  this  task  is 
described  in  [1].  Increasing  objects  can  reduce  the  distance  between 
objects  by  two,  then  we  are  sure  that  no  too  distant  objects  are  joined  and 
we  do  not  lose  any  k-separated  pixels,  because  earlier  we  have  checked  the 
problem  for  k=2,3. 

The  described  algorithm  can  be  realized  in  a  2-layer  DTCNN.  The  first  layer 
has  a  constant  template  for  increasing  objects.  The  second  one  is  a  layer 
with  cyclic  templates  (the  cycle  length  being  20).  The  first  8  snapshot 
templates  solve  the  problem  for  k=2,  the  next  12  for  k=3.  The  first  layer 
should  be  clocked  20  times  slower  then  the  second  one  and  the  clocks  roust 
have  different  phases.  Computation  of  the  result  for  the  first  layer  must 
be  completed  before  computation  in  the  second  layer  starts.  To  avoid  this 
and  make  the  timing  requirements  not  so  strong  we  can  add  one  snapshot 
template  to  the  second  layer  which  does  not  change  the  output  states  of  the 
cells  (a=l,  b=0,  i=0)  and  during  this  21'*  iteration  perform  one  iteration 
in  the  first  layer. 
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5.  CONCLUSIONS 


Using  presented  methods  we  are  capable  of  implementing  any  local  Boolean 
function  defined  on  the  r-neighbourhood.  For  that  we  need  a  single-layer 
DTCNN  with  time-variant  templates,  where  control  operator  is  defined  on  the 
r-neighbourhood  and  the  feedback  operator  consists  of  self-feedback  only. 
We  are  able  to  realize  both  the  sum-of-products  (method  1)  and  the 
product-of-sums  (method  2)  form  of  any  Boolean  function,  and  we  choose  the 
shorter  one.  These  methods  can  be  used  for  hexagonal  grids  without  any 
modif ications. 

Several  examples  are  demonstrated  and  discussed.  By  means  of  method  1  the 
problem  of  testing  minimal  distances  is  solved.  The  solutions  for  the  case 
of  the  ’maximum’  metric  and  k=l,2,3  are  described.  For  that  the  single 
layer  CNN  with  time-variant  templates  is  required.  A  two-layer  structure, 
which  solves  that  problem  for  arbitrary  value  of  k,  is  also  described.  For 
that  application  the  2 -neighbourhood  of  the  control  operator  and  the 
0-neighbourhood  of  the  feedback  operator  are  sufficient.  Similar  networks 
can  be  used,  when  searching  for  white  pixels  k-separated  by  black  ones.  The 
solution  for  that  problem  can  be  also  utilized  in  layout  design  for 
checking  is  there  are  any  too  thin  black  objects  on  the  grid.  The  solution 
for  the  Manhattan  metric  has  also  been  found. 
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Abstract 

This  papei  describes  the  nse  of  analog  CeOnlar  Nenral  Networks  (CNNs)  for  information  coding 
and  decoding  -  especially  for  the  case  of  moving  images.  The  dynamics  of  the  coding  (C-)  and 
decoding  (D-)  CNNs  are  described  by  geneialiaed  CNN  state  eqnations.  The  C-CNN  encodes  the 
image  by  structural  compression  and  halftoning.  The  D-CNN  decodes  the  received  data  through  a 
reconstruction  process  so  as  to  almost  recognize  the  original  inpnt  to  the  C-CNN.  The  importance  of 
our  compression  and  quantization  technique  lies  in  the  ability  to  make  the  computations  with  only 
local  connections. 

A  dynamic  quantisation  is  performed  in  the  C-CNN  to  decide  the  binary  value  of  each  pixel 
from  the  neighboring  values.  In  order  to  reduce  the  error  between  the  original  gray  image  and 
reconstructed  halftone  image,  the  template  synthesis  problem  is  addressed  from  the  viewpoint  of 
energy  minimization. 

The  structural  compression  template  synthesis  problem  is  discussed  from  the  viewpoints  of  topo¬ 
logical  and  regularization  theories.  The  structurally  compressed  image  is  regenerated  in  the  D-CNN 
by  a  dynamic  current  distribution. 

The  communication  system  in  which  the  C-  and  D-CNNs  are  embedded  contists  of  a  differential 
transmitter  with  an  internal  receiver  model  in  the  feedback  loop.  Examples  of  the  performance  of 
the  complete  system  are  given. 


1  Introduction 

The  Cellular  Neural  Network  (CNN)  first  proposed  by  L.  O.  Chua  and  L.  Yang  [1]  [2]  is  a  locally  intercon¬ 
nected,  large-scale  nonlinear  analog  neural  network  which  has  applications  to  many  practical  problems. 
A  generalization  of  the  CNN  paradigm  was  made  by  T.  Roska  and  L.  O.  Chua  which  covers  a  broad  class 
of  programmable  multidimensional  cellular  analog  processing  arrays  [3]  [4].  Of  special  interest  here,  it 
has  been  shown  that  the  nonlinear  dynamics  of  the  analog  CNN  can  perform  image  halftoning  [5]. 

1.1  System  Summary 

This  paper  describes  the  use  of  the  CNN  for  data  coding  and  decoding  for  the  purposes  of  lossy  fixed- 
rate  compression  and  quantization.  In  a  communication  system,  the  CNN  would  be  used  in  both  the 
transmitter  and  receiver.  It  will  be  shown  how  the  technique  can  be  applied  to  the  transmission  of  both 
still  and  moving  images. 

Two  distinct  compression  techniques  were  developed  which  only  require  local  connections  for  compu¬ 
tation.  As  shown  in  the  Figure  1,  the  coding  (C-)  and  decoding  (D-)  CNNs  implement  these  encoding¬ 
decoding  pairs. 

Two  types  of  compression  are  performed  in  the  C-CNN:  structural  compression  (S)  and  dynamic 
quantization  (H).  The  C-CNN  first  performs  the  structural  compression  which  locally  maps  the  input 
gray  image  to  a  smaller  gray  image.  The  main  work  of  the  C-CNN  is  a  dynamic  quantization  by  which 
the  gray  image  is  mapped  to  a  binary  image  of  the  same  size  for  digital  tnuismission.  The  dynamic 
quantization  is  done  by  halftoning. 

At  the  receiving  end,  the  D-CNN  attempts  to  regenerate  the  original  image  as  closely  as  possible.  The 
inverse  halftoning  operation  (H~^)  converts  the  received  binary  image  back  to  gray  scale.  The  decoding 
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Figure  1;  A  block  diagram  of  the  complete  communication  system  is  shown  with  the  functions  performed 
by  the  C-  and  D-CNNs.  The  dilTerence  between  the  current  acquired  image  u(n)  and  the  image  on 
display  at  the  recover  u*(n  —  1)  is  sent  to  the  C-CNN.  The  C-CNN  consists  of  two  functional  blocks, 
the  structural  compression  S  and  the  halftoning  H.  Halftoning  plays  a  dual  role  in  compression  and 
quantization  of  the  data  for  binary  transmission.  The  D-CNN  attempts  to  invert  the  halftone  H~^  and 
structural  compression  S~^.  Finally,  the  difference  image  is  added  to  the  one  currently  on  display. 


of  structural  compression  (S~^)  is  done  by  a  dynamic  current  distribution  which  maps  the  image  back  to 
the  original  dimensions. 

The  first  compression  coding-decoding  pair  is  structural  compression  and  the  corresponding  decoding 
done  by  current  regularization.  The  structural  compression  is  done  in  a  manner  similar  to  syndrome  error 
coding  [6].  The  novel  ciurent-type  CNN  is  used  to  perform  this  compression.  The  current-type  CNN  can 
be  written  as  a  special  version  of  the  CNN  equations. 

The  second  type  of  compression  is  dynamic  quantization  which  b  performed  by  halftoning  the  image. 
Halftoning  quantizes  each  pixel  of  the  image  to  a  binary  value.  Therefore,  halftoning  can  simultaneously 
be  considered  both  a  form  of  quantization  and  fixed-rate  image  compression.  In  the  usual  situation,  the  1 
bit/pixel  will  be  allocated  in  a  manner  that  will  be  a  trade-off  between  gray  level  and  edge  preservation. 
In  regions  of  the  image  with  slowly  changing  gray  levds  the  algorithm  will  try  tc  match  the  spatial  density 
of  bits  to  that  of  the  local  gray  level.  The  larger  the  constant  region  the  more  accurately  the  density 
can  be  matched.  However,  when  a  small  object  or  fine  structure  occurs  in  a  r^ion,  the  bits  in  that  area 
should  be  used  to  preserve  its  presence.  Doing  so  will  be  at  the  expense  of  accurate  preservation  of  the 
gray  level  of  the  object.  Halftoning  b  therefore  a  very  natural  type  of  compression  where  the  spatial 
information  density  b  assumed  to  be  constant. 

The  system  in  which  the  C-  and  D-CNNs  are  embedded  consbts  of  a  differential  transmitter  with 
a  local  receiver  model  in  the  feedback  loop.  When  transmitting  moving  images,  features  in  motion  are 
quickly  rendered  while  detail  accumulates  in  stationary  areas.  When  transmitting  a  still  image,  the  first 
&ame  will  omtain  a  recognisable  image  quickly  with  detsdl  and  quantbation  noise  reduction  supplied 
during  subsequent  transmissions. 

In  such  an  arrangement,  it  can  be  shown  that  the  error  of  successive  transmissions  does  not  accumulate, 
in  the  following  sense: 

Let  CDn  =  be  the  coding  and  decoding  done  on  image  n.  By  expanding  one  level  of 

recursion  in  the  block  diagram,  we  can  write 

u*(n  -H)  =  CP„+i(u(n  + 1)  -  u*(n  -  1)  -  C1?«(u(j»)  -  u’(n  -  1)))  +  u*(n  -  1)  -b  CP„(u(n)  -  u*(n  -  1)) 

Now,  if  the  last  transmission  was  perfect,  i.e.  CDn^^i  =  Identity,  then  u*(n  -b  1)  =  u(n  -b  1)  with  no 
accumulated  error. 

1.2  The  CNN 

A  Cellular  Neural  Network  (CNN),  as  first  proposed  in  [1],  b  a  continuous  time  neural  network  with 
diameter-limited  local  interconnections  and  a  unity-gun  piecewise  linew  approximation  to  the  standwd 
sigmoidal  output  function.  The  'neurons’  are  placed  in  a  regular  array  and  ‘syn^>tic’  connections  are 
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allowed  only  locally.  The  local  nature  of  the  interconnections  are  critical  when  considering  VLSI  imple¬ 
mentation. 

It  is  convenient  for  us  to  write  the  state  equations  in  convolution  form  [1]:  Let  u,  x,  y  be  the  M  x 
N  input,  state,  and  output  gray  scale  images  respectively,  which  are  assumed  to  be  zero  outside  of 
their  support.  Let  Ta  and  Tg  be  functions  such  that  Ta.Tb  ;  -+  1R  with  = 

and  (Ta),j,(Tb),j  =  0  for  l|(i,i)|loo  <  r,  some  small  neighborhood  radius.  Let 
I  be  a  real  constant.  The  symmetric  space-invariant  Cellular  Neural  Network  can  be  described  by  the 
following  state  and  output  equations; 

+  (Tx  ♦  y(0).j  +  (Ta  *  u),j  +  I  (1) 

y.j(0  =  /(*.j(0)  (2) 

where  f  :  x  >-*  |(|r  -f  1|  —  |r  —  1|),  ar  6  IR. 

The  functions  Tx  and  Tg  are  called  cloning  templates  and  can  be  considered  2r-|-l  x  2r  +  l  matrices. 
The  condition  (^x),j  =  is  called  template  symmetry.  An  even  stronger  condition  on  the 

templates  is  isotropy.  For  isotropy,  the  template  must  satisfy  (Tx),j  =  (7x)i  j  for  (i,j)  and  (k,l)  the 
same  Z-2  distance  from  the  origin.  This  condition  implies  template  symmetry  and  is  useful  in  halftoning 
applications. 

We  will  often  discuss  the  CNN  first  in  matrix- vector  form  [3].  Let  A  and  B  be  MN  x  MN  matrices 
representing  the  convolutions.  They  will  be  of  the  form  block  Toeplilz  with  Toeplitz  blocks.  Let  I  be  a 
real  constant.  The  Cellular  Neural  Network  can  be  described  by  the  following  slate  and  output  equations; 

^x(0  =  -x(0  +  Ay(0  +  Bu -f- /  (3) 

y(0  =  /WO)  (4) 


1.3  Outline  of  Paper 

In  Section  2  the  theory  behind  structural  compression  is  explained.  In  Section  3  the  specifics  of  halftoning 
and  inverse  halftoning  are  explained  from  the  viewpoint  of  energy  minimization.  Section  4  puts  the  two 
types  of  compression  together  into  CNN  equations.  Finally,  in  Section  5  we  present  the  techniques  and 
results  of  simulation  to  demonstrate  the  performance  of  the  end-to-end  system  when  using  moving  images. 


2  Structural  Compression 

The  first  type  of  local  compression  we  will  consider  is  structural  compression.  We  will  attempt  to  take 
advantage  of  spatial  redundancy  in  an  image  by  collapsing  the  image  support  onto  a  smaller  grid. 

2.1  Compression 

To  simplify  the  discussion,  consider  the  transmission  of  a  fixed  frame  of  a  still  image,  that  is,  u*(n— 1)  =  0. 
To  perform  structural  compression,  the  input  u(n)  is  ^plied  to  the  links  of  a  graph.  At  each  node,  the 
values  on  the  links  connected  to  that  node  are  summed  to  produce  the  output  B(n).  If  /  is  the  number  of 
links  and  r  is  the  number  of  nodes  in  the  graph,  the  compression  ratio  will  be  j. 

The  degree  of  compression  obtained  is  determined  by  the  topology  of  the  graph  used.  Graphs  which 
tile  the  plane  in  a  regular  way  are  the  most  convenient.  One  possibility  is  the  regular  square  grid,  as 
shown  in  Figure  2  which  has  has  two  links  for  every  node.  Another  is  the  hexagonal  grid  which  has  three 
links  for  every  two  nodes. 

It  is  straightforward  to  implement  this  method  with  a  circuit  by  only  using  local  connections,  as  shown 
'm  Figure  2.  On  each  link  of  the  graph,  a  current  source  is  placed  representing  the  input  at  that  location. 
In  practice  this  may  be  the  current  generated  by  a  photosensor.  Then,  by  KCL,  the  current  leaving  the 
node  on  the  output  line  is  the  sum  of  input  currents. 

Let  S  be  the  r  x  /  incidence  matrix  which  represents  the  compresuon  network.  Then,  the  compression 
can  be  represented  by  the  linear  transformation  B(n)  =  Su(n)  where  B(n)  is  the  syndrome  that  will  be 
sent  to  the  halftoning  network. 
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Figure  2:  A  small  section  of  the  encoding  (left)  and  decoding  (right)  circuits  for  structural  compression. 
In  the  encoding  network  the  input  is  mapped  onto  the  links  as  current  sources.  The  output  current  at 
each  node  is  sent  to  the  halftoning  network.  One  node  of  the  decoding  network  for  dynamic  current 
distribution  is  shown.  Intuitively,  the  network  is  reversing  the  process  of  encoding  by  iiyecting  the 
received  currents  into  the  resistive  grid.  The  resistors  ensure  that  the  least  norm  solution  is  obtained.  It 
is  clear  that  the  self  conductance  G  is  required  since  the  sum  of  the  received  syndrome  values  will  not  in 
general  be  zero  and  allows  for  some  current  to  leak  out.  The  output  is  the  current  through  the  resistors. 

2.2  Reconstruction 

The  problem  at  the  receiving  end  is  to  find  an  appropriate  inversion.  For  this  discussion,  we  assume  that 
the  halftoning  and  transmission  are  lossless  so  that  B*(n)  =  B(n) 

The  reconstruction  we  will  pursue  here  is  the  standard  pseudo-inverse  given  by  S^(SS^)~^.  This 
would  give  the  u*(ri)  with  least  norm  such  that  B*(n)  =  Su*(n).  Unfortunatly,  since  S  is  an  incidence 
matrix,  it  is  singular,  having  a  single  zero  eigenvalue  corresponding  to  the  constant  ugenvector. 

To  correct  for  the  singularity,  we  define  S~^  =  S^(G  -h  SS^)  where  G  is  a  small  multiple  of  identity. 

Since  SS^  is  positive  semi-definite,  G  -f  SS^  will  be  positive  definite  and  therefore  invertible. 

In  general,  the  pseudo-inverse  just  defined  is  not  represented  by  a  locally  connected  network.  However, 
consider  the  feedback  network  defined  by 

^x*(<)  =  -(G-bSS>*(t)-bB‘(n)  (5) 

u*(<)  =  S-^x*(0  (6) 

It  can  be  seen  to  have  the  appropriate  equilibrium  u*(oo)  =  S  ^B*(n).  Note  that  stability  is  assured  by 
the  positive  definiteness  of  G-b  SS^. 

Now,  this  is  exactly  the  dynamic  system  which  is  obtained  by  the  resistive  grid  shown  in  Figure  2. 
The  connections  are  all  local  and  of  the  same  form  as  the  compression  network.  The  system  could  also 
be  implemented  as  a  standard  CNN  with  connection  radius  2ra  -t- 1. 

3  Halftone  Compression  and  Quantization 

The  nonlinear  operation  performed  by  the  H-CNN  is  dynamic  gvanlualion  by  which  the  output  of  each 
neuron  becomes  a  binary  value.  Tliis  operation  can  be  formulated  in  the  same  manner  as  halftoning 
[5]  [7].  We  would  like  to  clioose  a  binary  image  y  which  when  reconstructed  by  a  filter  Q  looks  like 
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the  originid  image  8  filtered  by  R.  We  will  assume  that  Q  and  R  are  local  and  space-invariant.  It  is 
convenient  to  use  the  least  squares  criteria  as  a  measure  of  how  much  one  image  looks  like  another. 

We  will  use  the  CNN  to  attempt  to  solve  for  a  binary  output  y  to  minimize  the  generalized  least-square 
distortion 

dist(y,  8)  =  (Qy  -  R8)'^(Qy  -  Rb)  (7) 

The  desired  result  is  to  have 

Qy  »  Rb 

Let  8  be  the  input  to  a  CNN.  Now,  the  CNN  was  shown  in  [1]  to  always  decrease  the  Lyapunov-energy 
function 

^^W  =  -|y‘^(A-I)y-y^B8  (8) 

Let  A  be  the  matrix  representation  of  the  space-invariant  template  which  can  be  written  in  the  form 

A  =  -Q'^Q  +  diag(Q'^Q)  +  (1  -I-  f)I 

along  with 

B  =  Q^R 

Let  6  =  (Q^Q).,  j  the  value  along  the  diagonal. 

The  Lyapunov-energy  function  becomes 

m  =  -|y^(-Q‘"Q  +  (6  +  Oi)y  -  y'^Q'^Rs 

^(0  =  iy'^Q^Qy  -  y'^Q'^iu  -  -p  f)yTy  (9) 

The  -5(6+e)y^y  term  is  the  norm-increasing  term  which  ensures  that  the  steady  state  will  be  in  the 
saturation  region  giving  a  binary  output.  Note  that  this  term  is  a  constant  for  all  steady  state  outputs, 
so  there  is  some  reason  to  believe  that  it  has  little  effect. 

The  first  two  terms  are  equivalent  to  minimizing  the  distortion  as  defined  in  Equation  7  since 

di8t(y,  b)  =  (Qy  -  R8)^(Qy  -  Rs)  =  y^Q^Qy  -  y^Q^Rs  -  8''’R''’Qy  -|-  b^R''’R8 

The  input  b  is  a  constant  vector  so  minimizing  di6t(y,  b)  is  equivalent  to 

-  ^y'^Q'^Ra  -  ^s'^R'^Qy  =  ^y'^Q'^Qy  -  s'^R'^Qy  (10) 

So,  the  CNN  is  nearly  decreasing  the  chosen  distance  function  during  dynamic  operation.  Now,  since 
Qy  »  Rb  in  the  least  squares  sense  we  can  write 

B  w  R'^Qy 

so  that  we  can  approximately  recover  s  from  y  by  a  linear  filtering  operation  which  can  be  implemented 
by  a  D-CNN.  This  is  what  was  desired,  so  reconstruction  is  performed  by 

s*  =  R"*Qy 

As  a  special  case,  R  can  be  chosen  such  that  B  =  I  and  therefore  only  the  A-template  is  required  for 
coding.  Then  B  =  Q^R  =  I  so  R~^  =  Q^.  Therefore  the  reconstruction  can  be  done  by 

s*  =  Q'^Qy  (11) 

which  has  impulse  response  the  same  diameter  as  the  A-template  (in  fact,  Q^Q  is  the  negative  of  the 
A-template  with  the  diagonal  restored)  and  can  then  be  implemented  as  single  feedforward  template  in 
the  D-CNN. 

As  a  result,  the  B-template  is  not  required  for  halftoning  and  the  A-template  and  B-template  synthesis 
problems  can  be  discussed  $eparately  for  information  coding  and  decoding. 


4  Putting  it  all  Together 

We  will  now  show  how  the  compression  and  reconstruction  systems  just  described  can  be  combined  into 
the  CNN  framework.  From  the  block  diagram  and  the  discussion  in  Sections  2  and  3,  it  can  be  seen  that 
the  transmission  of  image  n  is  described  by: 

Uc(n)  =  u(n)  —  u*(n  —  1) 


C-CNN 


jix(t)  =  -x(0  +  Ay(t)  +  Suc(n) 

y(t)  =  /(x(0) 


ideal  channel  y*(n)  =  y(oo) 


D-CNN 


f  ^x*(t)  =  -Gx‘(t)  +  (-S)u4l)+Q-^Qy-(n) 

\  Ur,(i)  =  S'^s(x*(t)) 


u*(n)  =  Uo(oo)  +  u*(t»  -  1) 

The  relation  to  the  standard  CNN  is  clear.  The  C-CNN  fits  the  model  except  that  S  is  not  square  - 
which  is  a  minor  issue.  The  D-CNN  has  a  matrix  multiplication  at  the  output  which  does  not  match  the 
CNN  model.  Note  that  this  is  not  a  problem  for  the  state  equation  since  the  product  (— S)S^  represents 
a  standard  A-template.  Also,  it  may  be  necessary  to  scale  the  inputs  to  the  CNN  so  that  they  fall  in  the 
assumed  range. 

The  compression  acheivable  by  this  method  is  j  x  |  where  b  represents  the  standard  number  of  bits 
used  per  pixel  (often  b  =  8). 

The  function  g  was  chosen  to  be  linear  for  our  experiments.  However,  there  are  many  possibilities 
for  the  use  of  a  nonlinear  g.  For  example,  it  may  be  useful  to  guarantee  that  the  ouput  is  in  the  proper 
range  or  to  enhance  edges. 


5  Simulation  Results 

A  computer  simulation  was  developed  to  validate  and  demonstrate  our  results.  A  discrete-time  simulation 
of  the  differentiai  equations  was  used  to  simulate  the  C-CNN.  The  steady-state  of  the  D-CNN  was  arrived 
at  through  a  relaxation  algorithm. 

For  the  structural  compression,  the  square  grid  shown  in  Figure  2  was  used  with  unity  weighting.  The 
reconstruction  was  done  with  a  self  conductance  of  G  =  1/64  and  capacitance  (7  =  1. 

For  halftoning,  the  5x5  sampled  Gaussian  was  chosen  for  the  A-tempIate.  The  choice  is  based  on 
the  intuition  that  the  importance  of  errors  to  the  T'iewer  of  adjacent  cells  matters  less  with  distance.  The 
template  values  used  are  given  by 


Tac  = 


0.0000 

-0.0104 

-0.0208 

-0.0104 

0.0000 


-0.0104 

-0.0625 

-0.1042 

-0.0625 


-0.0208 

-0.1042 

1.0500 

-0.1042 


-0.0104  -0.0208 


-0.0104 

-0.0625 

-0.1042 

-0.0625 

-0.0104 


0.0000 

-0.0104 

-0.0208 

-0.0104 

0.0000 


Tao  = 


0.0000 

0.0104 

0.0208 

0.0104 

0.0000 


0.0104 

0.0625 

0.1042 

0.0825 

0.0104 


0.0208 

0.1042 

0.1667 

0.1042 

0.0208 


0.0104 

0.0625 

0.1042 

0.0625 

0.0104 


0.0000 

0.0104 

0.0208 

0.0104 

0.0000 


The  system  was  used  to  encode  and  decode  successive  images  of  a  woman  in  motion.  The  results 
of  applying  the  CNN  coding  and  decoding  are  shown  in  Figure  3.  For  the  experimental  system,  the 
compression  rate  achieved  was 
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Figure  3:  The  15th  original  image  and  the  (50th,  100th,  149th)  regenerated  images. 
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Abstract 

In  this  paper  we  consider  mathematical  model  of  multi-valued  neural  element 
which  is  suggested  to  be  used  as  one  of  the  alternatives  of  the  basic  CNN 
element.  On  the  basis  of  such  CNN  we  offer  the  model  of  the  associative 
memory  for  storing  grey-scale  images.  Efficient  learning  algorithm  for 
neural  element  and  associative  memory  is  presented  in  the  paper. 

I.  INTRODUCTION 

CNN,  introduced  in  [1]  and  having  been  extensively  developed  recently  [2  and 
others],  became  brilliant  alternative  to  conventional  computers  for  image 
processing  and  recognition. 

Specifically,  on  the  basis  of  CNN  were  introduced  a  number  of  models  of 
associative  memory  and  were  developed  the  methods  of  solving  the  problems  of 
image  recognition  and  filtering  [2  -  4].  Despite  the  fact  that  a  lot  of 
complicated  tasks  are  solved  on  the  basis  of  CNN,  still  we  assume  that  in  a 
great  number  of  papers  the  range  of  processed  signals  is  "unfairly"  restricted 
-  either  mostly  binary  (bipolar)  or  exclusively  binary  signals  (in  case  of 
associative  memory  designing)  are  considered.  To  break  this  restriction  and 
extend  both  CNN  functionality  and  the  range  of  problems  that  are  solved  on  CNN 
basis,  we  suggest  using  neural  element  based  on  the  mathematical  model  of 
multi-valued  threshold  element  [5]  as  a  basic  ,  CNN  neuron,  as  well  as  an 
alternative  to  neurons  considered  in  [i,  6].  Proceeding  from  the  idea  that 
application  of  this  element  is  natural  for  multi-valued  signals  processing  and 
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considering  the  quickly  convergent  learning  algoritlim  (presented  below),  we 
believe  that  application  of  this  neuron  can  be  highly  efficient.  For  instance, 
designing  cellular  associative  memory  for  storing  grey-scale  images  and 
implementation  of  image  filtering  algorithms  on  such  CNN  is  absolutely  natural. 
Another  advantage  of  this  basic  CNN  neuron  is  convenience  of  its  analog 
(including  optical)  hardware  implementation,  since  operations  with  complex 
numbers  provide  "arithmetical  basis"  of  the  neuron. 


11.  MULTI-VALUED  CNN  BASIC  ELEMENT 

To  simplify  the  analysis,  we  will  consider  discrete  time  version  of  the  CNN, 
though  it  is  evident  that  all  the  results  presented  below  are  easily  trasformed 
to  the  continuous  time  case. 

Suppose  we  have  CNN  of  a  dimension  N‘M,  We  will  apply  multi-valued  neural 
elements  as  basic  neurons  of  this  network.  Each  of  these  elements  performs  the 
following  transformation: 

Yjj(t+1)  =  F  [  Wg  +  D  w‘J  x‘J(t)  1.  (  1  ) 


where  Y,  .  -  neuron  state,  connection  weights  corresponding  to  m-th 

ij  mm  w  • 

input  of  ij-th  neuron  and  input  signal  value  on  m-th  input  of  ij-th  neuron 
respectively.  F(')  -  output  function  of  ij-th  neuron  that  will  be  defined 
furtheron. 

Let  input  signals  X  and  output  signal  Y  for  each  neuron  be  located  in  the 
range  0,  k-1,  i.e.  each  neuron  at  each  particular  moment  performs  some 

function  of  k-valued  logic  determined  by  weights  W.  It  is  evident  that  for  byte 
images  k  =  256. 

Let’s  assume  that  each  value  of  0,  ...»  k-1  corresponds  to  complex  number  r^, 
j  =  0,  1,  k-1  like  this: 


Tj  =  €xp(i  -2%  j/k)  (  2  ) 

where  i  is  an  imagenary  unit.  -,E.g.,  ~  1.  =  £»  primitive  k-th  power  root  of 

2  k-1  ^  ^ 

a  unit,  r2  =  £  ,  rj^_j  =  £  .  Following  this  input  and  output  signals  for 

each  neuron  will  be  coded  by  k-th  power  roots  of  a  unit. 

Let  a/g(Z)  -  argument  of  complex  number  Z  (  0  ^  <  2%  ).  Now  we  can 

determine  output  function  F  for  neuron  (refer  to  (1)).  Let’s  define  function 
CSIGNCZ),  Z  ^  C  (C-Complex  Numbers  Field)  as  follows: 
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if  2%'\/k  <  a/i'(Z)  <  27C(j+l)/k, 


CSIGN(z)  =  exp(i-29U  j7k)  =  £  , 

CSIGN(O)  =  £°  =  1.  (  3  ) 

The  interpretation  of  the  (3)  may  be  the  next.  If  complex  plane  is  separated 
into  k  equal  sectors  and  complex  number  Z  is  located  in  j-th  sector  then 

function  CSIGN(Z)  equals  (  j  =  0,  1,  k-1;  £  is  a  primitive  k-th  power 

root  of  a  unit ) . 

Specifically  CSIGN  function  will  be  used  as  output  function  F  of  the  neuron. 
In  this  case  (1)  will  be  transformed  to: 

Y,  (t+1)  =  CSIGN  [  W„  +  2  W‘J  x‘J(t)  ].  (  4  ) 

m 

So,  (4)  describes  dynamics  of  the  ij-th  neuron.  Naturally,  since  Y  and  X  are 
complex  numbers  then  weights  W  are  complex  too.  In  specific  case  when  k  =  2  and 
weights  are  real,  (4)  will  describe  linear  threshold  element  and  according  to 
(3)  CSIGN  will  become  natural  function  SIGN. 

HI.  LEARNING  ALGORITHM  FOR  MULTI-VALUED  NEURON 

Further  we  will  describe  learning  algorithm  for  multi-valued  neural  element. 
In  the  case  k  =  2  the  problem  is  reduced  to  the  well-known  problem  of 
Rosenblatt’s  Perceptron  learning.  Here  we  will  consider  the  case  when  k  ^  2, 
paying  special  attention  to  the  case  k  >  2. 

Let  k  ^  2  be  an  arbitrary  natural  number.  Let’s  have  k  of  non-intersecting 
learning  subsets  ^j~  •••»  }•  J  ”  •••»  ^  “  (^0’  ^1’  *'■’ 

X^),  where  Xq  =  1,  and  other  coordinates  take  their  values  from  the  set  {£^,  £^, 
k-1 

£  }.  When  Aq . ^k-1  predetermined,  the  task  of  learning  consists 

in  figuring  out  of  permutation  (OCq,  CX^,  ....  vector  W  =  (W^,  W^, 

Wn)  satisfying 

CSIGN(X,  W)  =  £  J  (  5  ) 

for  each  X  £  Aj,  j  =  0,  1,  ...»  k-1,  where  W  =(Wq,  w^,  w^)  -  vector 

complex-conjugated  to  W;  (X,W)  =  Wq  +  +  ...  +  w^x^  -  scalar  product  of 

(n+l)-dimensional  complex-valued  vectors  in  (n+l)-dimensional  space.  If  (5)  is 
true,  sets  Aq,  A^,  ...,  Aj^_j  are  called  edge-separable. 

Since  we  consider  neural  network,  we  can  assume  that  permutation  (a^,  Ot^, 
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••T  ^-j)  known,  because  it  will  be  determined  by  the  state  of  neurons  from 
nearest  neighbourhood  of  given  neuron. 

Now  we  have  to  make  iterative  evaluation  of  sequence  of  weight  vectors 

Wq,  Wj,  ....  for  it  to  satisfy  condition  (5)  starting  from  some  number  m  and  = 


w 


m+1  * 

The  sequence  is  evaluated  as  follows: 

a 


W  ^  =  W  +  U  -D  •  £  , 

m+1  m  s  m  m 

m 


(  6  ) 


where  -  correction  coefficient,  is  equal  0,  1,  2  or  3  depending  on  one 

of  the  following  cases: 


a).  Wq  =  0,  if 


b).  Wj  =  -i£,  if 


CSIGN(X_^,  W^)  =  £“+ 


CSIGN(X„,  W^)  =  e 


a.+i 

j 


for  k  =  2,  3  and 


a,+l  _  a.+  tk/4] 

£  -<  CSIGN(X^,  Wj^)  ^  for  k  ^  4; 


c).  W-  =  1,  if 


a,+[k/4i+i  _  a,+3[k/4]-i 

£  J  CSIGN(X^.  W^)  ^  £  J  for  k  ^  4; 


d).  W3  =  i.  if 


CSIGN(X^,  W^)  =  £ 


aj.2 


for  k  =  3  amd 


a,+3[k/4]  _  a.+k-l 

;  J  -{  CSIGN(X^,  W^)  ^  £  J 


for  k  ^  4. 


Here  [k/4]  is  an  integer  part  of  k/4,  "  -<  "  is  defined  as  follows: 

— >  p  ^  q  (mod  k). 


£P  -<  £^ 


Let’s  clarify  the  meaning  of  rule  (6).  Note  that  in  our  case  D  =  D  =  1, 

ID 

m  =  0,  1,  ...,  because  all  the  components  of  all  input  vectors  X  have  a  constant 

modulo  (magnitude)  equal  to  1.  In  case  a)  vector  W  =  W  satisfies  (5)  when  X  = 

m 

X^,  so  there  is  no  reason  to  change  it.  In  cases  b)  -  d)  weight  vector  is 

corrected  by  component-by-component  addition  to  W  of  some  vector  depending  in 

-a  "* 

~  n 

each  case  of  the  difference  between  £  '^CSIGN(X  ,  W  )  and  £  .  The  second 

m  m' 
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addendum  in  (6)  is  evaluated  from  following  reason: 

OC. 

(X  ,  w  .)  =  (X  .  wj  +  (X  .  Q  •£  J-xj  = 

'  m  m+1'  'm  m'  'ms  m' 

m 

a.  _  a 

=  (X  ,  W  )  +  W  •£  J(X  ,  X  )  =  (X  ,  W  )  +  (n+lXo  •£  J  , 

^  m  m'  ^  '  m’  m'  ^  m  ’  m'  '  s  ’ 


where  n  is  number  of  neuron  inputs.  OJ  is  introduced  to  make  the  value  of 

m 

expression  ^ 

CSIGN{X^,  =  CSIGN  [(X„,  W_^)  +  (n+l)0^  £  •>  ] 

a. 

closer  to  £  at  each  next  iteration. 

Initial  value  for  vector  W  can  be  arbitrary  (furtheron  we  will  consider 
its  selection  for  associative  memory  learning). 

So  iterative  learning  process  i>  reduced  to  sequent ional  application  of 

formula  (6)  taking  into  account  rules  a)  -  d)  till  case  a)  is  reached.  It  is 

evident  that  if  (X.  is  fixed,  this  process  is  converged  maximum  on  step  5-6.  If 
^  CL 

permutation  (ttQ,  Otj,.-,  ^-i)  fixed,  then  Wq  is  the  first  to  be  evaluated 

according  to  (6).  Then  we  use  the  obtained  vector  for  evaluation  of  Wq  ,  etc.  , 

every  time  checking  new  weight  vector  for  previous  Otj.  This  procedure  goes  on 

until  vector  satisfies  (5). 

m 

THEOREM  I.  (On  convergence  of  learning  algorithm).  If  learning  subsets  A^, 
Aj,  ...,  Aj^_^  are  edge-separable,  then  it  will  take  finite  number  of  steps  to 

obtain  vector  satisfying  (5)  for  all  learning  subsets. 

If  we  presume  that  learning  process  is  infinite  then  we  will  obtain 
contradiction  with  edge-separation  of  learning  subsets. 

So  investigation  of  the  possibility  of  neuron  learning  in  each  particular 
case  is  reduced  to  the  investigation  of  the  edge-separation  of  learning  subsets. 
Note  if  this  issue  is  settled  positevely,  iterative  learning  process  is 
converged  pretty  fast.  The  number  of  iterations  can  reach  hundreds,  sometimes  - 
thousands,  but  not  more. 

IV.  MODEL  OF  ASSOCIATIVE  MEMORY  ON  THE  BASE  OF  CNN  FROM 

MULTI-VALUED  NEURONS 


For  solving  the  task  of  storing  m  patterns  (grey-scale  images)  determined  by 
12  m 

matrixes  P  ,  P  ,  ...,  P  of  N  *  M  dimension  and  each  image  contains  k  of 
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grey-scales;  0,  1,  k-1,  let’s  use  CNN  based  on  multi-valued  neurons.  The 

task  is  reduced  to  evaluation  of  such  weights  W  that  any  of  patterns  could  be 

reconstructed  during  finite  number  of  steps  from  the  input  pattern  that 

differs  from  P^  in  a  number  of  points. 

So  let’s  recode  given  patterns  P  from  integer-valued  to  complex-valued 

(0  — >  •••.  (k-1)  — >  Then  for  evaluation  of  the  connected  weights 

between  ij-th  and  pq-th  neurons  we  will  use  the  following  Hebb  rule  [3] 
generalization  for  complex-valued  patterns: 


W°. 

ij.pq 


0  for  Ii-Pl  >  1  or  |j-q|  >  1 

^  ^  ^  ^ 


where  P^^  is  an  ij-th  point  of  1-th  input  pattern,  “  "  is  the  complex-conju¬ 
gation. 

Having  evaluated  all  weights  for  all  the  neurons,  we  obtain  initial  value 
for  weights  of  our  associative  memory.  Further,  for  evaluation  of  final  values 
of  weights  W  that  will  determine  dynamics  of  the  associative  memory,  it  is 
necessary  to  apply  learning  algorithm  based  on  (6)  for  each  neuron  from  CNN.  It 
is  evident  that  the  speed  of  learning  algorithm  operation  will  be  the  higher  the 
more  patterns  P  are  correlated  among  themselves  or  ,  which  is  the  same  in  our 
case,  the  less  is  the  Euclid  distance  between  patterns. 
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Abstract: 

In  this  contribution  the  problem  of  cellular  neural  network  parameter  tolerances  is  dis¬ 
cussed  with  special  regard  to  the  network  generalization  abilities. 


1.  Introduction 

In  the  contemporary  state  of  art  of  the  theory  of  eirtificial  neural  networks  in  gene¬ 
ral  and  also  of  cellular  neural  networks  especially  a  good  part  of  attention  is  given  to 
the  problems  related  to  efficient  metnods  of  network  learning  and  to  the  questions  con¬ 
cerning  the  neural  network  approximation  abilities.  The  people  dealing  with  the  cellular 
neural  network  realization  give  a  lot  of  attention  to  finding  of  structures,  which  can  be 
implemented  to  VLSI  chips  and  slices. 

Very  few  of  these  authors  have  payed  their  interest  to  one  very  fundamental  problem, 
which  is  the  influence  of  designed  artificial  neural  network  parameters  deviations  from 
their  theoretically  optimal  (nominal)  values,  found  by  the  use  of  theoretical  procedures 
of  network  synthesis,  on  the  overall  cellular  network  function. 


2.  Tolerance  Problems 

In  the  general  system  theory,  already  many  years  the  problem  is  known,  how  to  opti¬ 
mize  the  nominal  system  design  so  that  the  influence  of  these  parameter  changes  will  be 
minimized.  The  optimization  is  usually  made  so  that  the  fundamental  (primary)  system 
properties  (e.g.  their  transformation  functions)  are  conserved  while  some  of  the  secondary 
properties,  e.g.  the  parameter  tolerances  are  optimized.  In  the  space  of  parameter  tole¬ 
rances  this  means  usually,  that  such  tolerance  polyhedra  is  searched,  which  involves  the 
maximal  amount  of  acceptable  systerr  realizations.  The  solution  of  this  problem  is  easy  in 
the  trivial  cases  only,  where  the  straightforward  methods  of  system  analysis  are  applicable. 
However,  if  the  number  of  system  parameters  is  slightly  higher  (several  few  tenths),  such 
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approach  is  to  be  replaced  by  the  considerably  complicated  numerical  procedures  based 
usually  on  statistical  methods.  The  respective  computation  is  often  complicated  and  time 
consuming  and  therefore  a  lot  of  human  mental  energy  was  given  to  the  search  for  effective 
and  efficient  design  optimization  methods. 

The  solution  of  the  tolerance  optimization  problem  for  neural  networks  in  general  is  not 
solved  up  to  now.  Nevertheless,  some  attempts  in  this  respect  were  published  elsewhere 
(see  [1,2]). 

When  dealing  with  cellular  artificial  neural  networks,  we  face  to  a  little  better  situation. 
While  the  complexity  of  the  mentioned  problem  is  straightly  related  to  the  number  of 
optimized  parameters,  we  can  use  the  advantage  of  the  local  connections  existing  between 
individual  neuronal  cells.  Therefore,  the  wave  matrices,  with  which  we  have  to  deal  are 
much  more  spare.  Their  optimization  with  respect  to  the  influence  of  parameter  deviations 
Is  solvable  with  much  less  effort  and  also  with  much  less  powerful  computer  tools.  For  this 
some  of  the  methods,  developed  on  the  base  of  statistical  derivatives  and  design  centering 
are  applicable. 

In  cellular  neural  networks  however,  some  other  problems  appear: 

•  one  of  them  is  the  optimization  of  the  structure  as  whole,  i.e.  the  solution  of  the 
optimal  connection  between  the  eventually  nonuniform  connected  neural  cells  or 
their  nuclei; 

•  second  problem  consists  in  the  fact,  that  the  sensitivity  of  network  properties,  i.e.  of 
the  network  transformation  function  depends  on  the  number  of  network  parameters 
(i.e.  weights  in  this  case)  and  decreases  with  the  redundancy  of  the  structure  in 
general.  Therefore,  when  dealing  with  cellular  neural  networks  one  can  expect 
higher  sensitivity  with  respect  to  cell  parameters. 

Also  this  kind  of  problems  are  principally  solvable  through  design  centering.  The 
design  procedure  can  be  applied  here  in  two  ways: 

1.  on  the  unlearned  structures  as  a  tool  for  final  fine  network  training,  where  it  often 
helps  to  reach  lower  error  values  than  obtainable  by  the  use  of  standard  learning 
procedures  only  (of  course,  according  our  experience,  the  starting  from  some  pre- 
leamed  point  in  the  respective  multidimensional  space  is  of  the  advantage), 

2.  on  the  satisfactory  trained  structures  as  a  tool  for  finding  such  solution,  which 
gives  better  properties  of  the  secondary  category,  e.g.  with  respect  to  generalization 
ability. 

Generalization  in  general  represents  the  ability  of  the  network  to  give  the  expected 
response  also  on  inputs  coming  from  regions,  which  were  not  trained.  If  the  term  “ex¬ 
pected”  is  reasonable  defined,  one  of  the  main  questions  is  to  find  the  minimal  training 
regions,  which  are  satisfactory  for  to  reach  the  requested  output  in  other  regions  (see  [3] 
e.g.).  The  analysis  of  parameter  tolerances  allows  us  to  attack  this  question  from  another 
side,  i.e.  to  try  to  find  the  value  of  influence  of  weight  matrices  for  which  the  changes 
of  investigated  secondary  network  properties  do  not  change  more  than  the  prescribed  £ 
value. 
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From  such  an  analysis  for  individual  cells  we  caji  made  some  estimation  on  also  for 
the  whole  cellular  neural  structure  (e.g.  through  expansion  in  series  of  nonlinear  func¬ 
tions  in  the  set  of  given  points  in  the  training  regions).  In  the  case  of  cellulax  neural 
networks  the  expected  accuracy  of  such  an  estimation  can  be  considerably  good,  because 
the  respective  nonlinear  transformation  is  applied  only  once  (for  one  layer  network)  and 
the  mathematical  expressions  of  the  expanded  series  axe  related  on  sparse  weight  matrices 
only. 
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Abstract 

Signal  processing  in  Cellular  Neural  Networks  (CNNs)  consists  of  multiplying  and  adding 
of  signals,  of  calculating  correlations  and  of  shaping  signals  with  a  nonlinear  fimction.  These 
operations  are  transformed  into  optical  processing  steps  by  using  spatial  light  modulators  (SLMs) 
and  lenses  as  basic  optical  elements.  The  SLM  performs  multiplications  massively  in  parallel  and 
with  the  speed  of  light  whereas  lenses  realise  a  Fourier  transform.  A  special  system  with  two 
SLMs  and  two  lenses  is  able  to  represent  an  array  of  CNNs  without  severe  limitations  on  the 
interconnections  between  neurons.  The  paper  gives  an  introduction  and  an  overview  on  optical 
signal  processing  for  CNNs.  It  is  concluded  by  an  example  for  optical  pattern  recognition. 

1  Introduction 

Analog  and  digital  signal  processing  performed  electronically  is  a  well  established  art  as  all  necessary 
components  are  available.  For  all  practical  cases  they  are  even  avadable  in  miniaturized  or  integrated 
form.  Therefore  other  physical  menus  of  processing  such  as  optical  ones  should  only  be  considered 
if  they  offer  convincing  advantages.  One  is  inclined  to  go  optical  for  the  enhancement  of  speed  of  a 
single  operation  since  it  may  be  performed  very  fast  with  the  speed  of  light.  However,  as  we  shall  see 
later,  for  many  applications  this  may  not  be  the  overriding  advantage  of  optical  processing,  especially 
because  one  has  to  consider  the  time  for  a  sequence  of  operations.  We’ll  investigate  the  issue  for 
Cellular  Neural  Networks  (CNN’s)  introduced  by  L.  Chua  [1]  [2].  After  recalling  the  basic  equations 
and  circuits  for  Chua’s  CNN’s  we  shall  have  a  comparative  look  at  the  electronic  and  optical  processing 
required.  This  is  followed  by  a  more  detailed  description  of  the  components  and  systems  for  optical 
signal  processing.  The  presentation  will  be  concluded  by  an  example  for  optical  CNN’s. 

2  The  Cellular  Neural  Network 


In  electronic  realizations  CNNs  are  as  a  rule  only  connected  with  their  direct  neighbors.  An 
interaction  with  more  neighbors  is  desirable.  If  N  neurons  were  fully  interconnected  we  had  for  each 
neuron  N-1  incoming  lines  from  other  neuions  in  addition  to  one  line  from  its  own  feedback  resulting 
in  interconnections  among  all  neurons.  Even  for  moderate  numbers  of  N  the  interconnections 
would  be  no  more  feasible  on  an  IC  even  not  with  several  layers  of  metalization. 
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The  signal  processing  within  a  neuron  with  the  architecture  in  figure  1  is  described  by  the  state 
equation 

T  T  T  T 


a:<,j(n+  1)  = 


fc=:  — rZ=  — T  1:=:— rl=  — r 


(1) 


which  represents  analog  sampled  data  processing  at  the  discrete  times  n,  n  +  1  etc.  .  It  could  also  be 
formulated  for  digital  processing.  The  terms  in  equation  (1)  are; 


x,j(n  -1- 1) 

the  state  of  neuron  i,  j  at  the  time  slot  n  -)- 1 

the  feedback  operator  (A-template) 

the  output  of  neuron  t,  j  at  the  time  slot  n 

Ki 

the  control  operator  (B-template) 

the  input  at  neuron  i,  j 

dij 

a  bias  which  shifts  the  nonlinear  function 

r 

the  size  of  the  neighborhood 

[3]; 


Reciprocal  time-discrete  CNN’s  with  arbitrary  sigmoid  functions  /  converge  to  their  proper  output 


W.j(«)  =  /(a:i,i(n)) 


(2) 


The  signal  processing  required  for  CNNs  is  as  follows: 


i)  multiplications  of  the  signals  yi^j  and  tq-j  with  the  weights  a*  ^  and  in  the  templates  for  which 
operational  amplifiers  with  a  network  of  resistors  can  be  used;  in  digital  realizations  expensive 
multipliers  are  necessary.  The  multiplications  should  preferably  be  carried  out  in  parallel  yielding 
a  large  number  of  multipliers. 

ii)  Equation  (1)  contains  two  crosscorrelations  between  the  weights  in  the  templates  and  the  signals 
for  which  again  adders  are  needed.  Electronically  this  again  is  performed  by  operational  ampli¬ 
fiers  with  resistors  for  the  adding.  Adding  comes  relatively  easy  for  analog  or  digital  circuits  if 
it  is  done  electronically. 

iii)  The  three  terms  in  equation  (1)  have  to  be  added. 

iv)  The  nonlinearity  /  again  is  realizable  by  an  operational  amplifier  or  by  a  network  of  biased 
diodes.  The  shape  of  /  can  be  formed  relatively  free  and  easy  by  electronic  means. 

v)  In  digital  CNN’s  delay  must  be  inserted  in  the  feedback  path  in  order  to  avoid  nonrealizable 
delay-free  loops. 

vi)  For  a  large  size  of  r  of  the  neighborhood  the  interconnections  would  lead  to  a  large  number  of 
metallization-layers  on  an  IC.  Practical  limit  presently  are,  however,  three  layers. 


Next  we  have  to  look  at  properties  of  optical  components. 

3  The  spatial  light  modulator  (SLM) 

The  liquid  crystal  (LC)  flat  panel  display  'mown  from  TV-systems  is  one  of  the  basic  elements  for 
optical  processing.  Firstly  its  operation  is  outlined  before  we  turn  to  the  specifics  of  optical  processing. 
The  LC-cells  in  figures  2a)  and  2b)  contain  the  LC-material  usually  of  the  twisted  nematic  type 
embedded  between  two  orientation  layers  on  two  glass  plates.  Between  the  two  transparent  electrodes 
underneath  the  orientation  layers  a  voltage  U  can  be  applied.  If  no  voltage  is  applied  as  in  figure  2a) 
the  lengthy  molecules  of  the  LC  orient  themselves  in  a  helix  where  the  groves  of  the  orientation  layers 
hold  the  molecules  on  each  side  in  a  given  position  oriented  perpendicular  to  each  other.  Because  of 
the  helix  the  material  is  called  twisted  nematic.  Incoming  light  is  linearly  polarized  by  a  polarizer-foil. 
The  structured  arrangement  of  the  LC-molecuIcs  provides  an  electrical  and  optical  anisotropy  of  the 
LC-material  with  speed  of  light  depending  on  the  direction  of  the  light.  This  can  be  explained  by  a 
different  speed  of  the  ordinary  and  the  extraordinary  beam.  The  effect  is  called  birefringence  and  is 
the  cause  for  a  rotation  of  the  polarized  light  by  90°  if  the  thickness  d  of  the  LC-material  is  chosen 
properly.  The  analyzing  polarizer  is  oriented  in  the  same  direction  as  the  incoming  polarizer.  Thus 
polarized  light  turned  by  90°  cannot  pass  the  second  analyzer.  The  cell  appears  dark.  Passing  of 
light  is  blocked.  If  a  sufficiently  large  voltage  U  is  applied  as  in  figure  2b),  the  LC-molecules  orient 
themselves  in  parallel  to  the  electrical  field.  In  this  mode  no  rotation  of  the  polarization  occurs  and 
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Figure  3:  Tlansnussion-vetsus-voltage  characteristic  of  a  TN-cell 

the  largest  picture  frequency  presently  available  with  TN-  cells.  The  time  ti  represents  the  speed  bot¬ 
tleneck  in  optical  processing.  Till  now  electronically  addressed  SLMs  with  up  to  575  x  2088  =  1200600 
picture  elements  each  with  a  size  of  130  x  48/tm^  have  been  produced  in  our  laboratory  [6]. 


Figure  4:  Addressing  circuit  of  a  pixel  a)  electronical  b)  optical 


The  optical  addressing  of  an  LC-cell  is  depicted  in  figure  4b).  An  incident  light  beam  hits  a 
photoconductive  layer  and  lowers  its  resistance  JSph  doe  to  the  intensity  of  the  light.  This  changes  a 
resistive  voltage-divider  such  that  the  voltage  across  a  pixel  of  the  LC-cell  increases  yielding  a  higher 
transmissivity  of  the  LC-material.  The  cell  figure  4b)  has  to  be  operated  in  a  reflective  mode  since 
the  photoconductor  is  not  transparent.  The  light  to  be  modulated  is  shone  onto  the  cell  from  the  same 
side  as  the  reflected  beam  emits. 

4  Optical  components  of  signal  processing 

The  components  are  treated  in  the  same  sequence  as  in  paragraph  2. 

4.1  The  optical  multiplication 


The  liquid  crystal  display  or  an  SLM  in  figure  5  exhibits  a  2-dimensional  array  of  pixek  at  the 
location  Xj ,  each  of  which  has  an  electronically  or  optically  controlled  transmission.  We  assign  a 
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pixel  to  each  neuron.  If  we  place  as  input  signals  a  picture  U {xi,  y,)  with  given  grey  shades  Ui,j  in  the 
N  pixels  Xi,  yj  behind  the  SLM  A{xi,  yj)  in  figure  5  with  also  N  pixels  exhibiting  the  grey  shades  Oi^j 
and  shine  light  through  both  then  the  outgoing  light  intensity  represents  the  analog  multiplication  [7] 

0{xi,yi)  =  U{xi,yj)A{xi,yj)  (®) 

as  each  light  beam  through  a  pixel  Xi,yj  is  attenuated  by  the  equation 

OiJ  =  (6) 

SLM’s  with  N  >  10^  are  presently  fabricated.  Therefore  the  set-up  in  figure  5  realizes  more  than  a 
million  analog  multiplications  in  parallel  and  with  the  speed  of  light.  This  massive  and  fast  parallelism 
is  one  of  the  biggest  assets  of  optical  processing. 

4.2  The  optical  crosscorrelation 


SLM  Lens  Filter  Lens 

(input  display  and  A(a»»,Wy) 

nonlinear  element) 


Figure  6:  An  optical  Time-Discrete  CNN 

It  is  known  in  optics  [7]  that  the  first  lens  in  figure  6  with  focal  length  /  performs  a  2-dimensional 
Fourier  transform  of  a  picture  or  a  field  in  the  front  focal  plane  with  the  complex  light  wave  amplitude 
Y{xi,yj).  The  Fourier  transform  is  the  spectrum 

-OP  -OP 


//  y(*.,  jfclexp  (-i(u>,a!j  Wyjfi))  djfc 


with  the  spatial  frequencies  in  x-  and  in  y-direction 


2  •  T 

w,  - 


and  w,  = 


as  well  as 


j  the  imaginary  unit  j  =  \/M 
A  the  wavelength  of  the  used  light  wave 


The  scaling  factor  in  front  of  equation  (7)  is  independent  of  the  variables  and  is  therefore  from  now 
on  neglected.  The  equation  (7)  is  an  approximation  due  to  Fresnel  which  only  holds  for  thin  lenses 
and  paraxial  light  rays. 
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Now  we  turn  to  a  4-f  system  which  consists  of  the  two  lenses  in  figure  6.  The  spectrum  Z(«i,a»y) 
in  equation  (7)  is  located  in  the  back  focal  plane  of  the  first  lens  where  it  is  multiplied  by  A{uix,Uy) 
using  an  SLM.  The  bar  stands  for  the  corrugate  complex  of  A.  The  spectrum 

S{Wx,Uly)  =  Z{u^,Uy)  A{Wx,Uy)  (g) 

at  the  output  of  the  SLM  is  again  Fourier-transformed  by  the  second  lens  yielding 

oo  oo 

J  j  S(u^,Uy)exp(-j{u^u,+Uyu'y)^dw^dMy  (9) 

—  OO  — oo 

We  now  have  to  clarify  the  meaning  of  O  in  the  new  plane  of  spatial  frequencies  w*  and 

u'y.  An  inverse  Fourier- transform  is  not  available  with  lenses.  Since  we  are  accustomed  to  the  result  of 
the  inverse  Fourier-transform  leading  from  the  domain  of  spatial  &equencies  back  to  the  space-plane 
we  look  at  the  space-plane  expression  corresponding  to  5(b>xi<^y),  namely 

OO  OO 

0(x,  y)  =  II  S(w*,wy)exp(j(w*x  +  wyy))d«x  duy  (10) 

—  OO  ^oo 

A  comparison  of  equations  (9)  and  (10)  reveals  that  the  result  O'  obtained  by  the  second  lens 
coincides  with  O  by  interpreting  the  variables  and  u>y  as 


"v  =  -y 


yielding 


0'{  x,-y)  =  0(*,y) 


Therefore  the  mirror-image  of  O  is  identical  to  the  inverse  Fourier-transform  of  5(a;x,a^y)  in 
equation  (8).  On  the  other  hand  the  inverse  Fonrier  transform  in  equation  (  10)  crm  also  be  calculated 
by  a  convolution  of  the  inverse  Fourier  transforms  ao(x,y)  ofA{u>x,Vy)  and  y(x,y)  of  Z(w*,u>y),  where 
ao(z,  y)  is  the  point  spread  function  of  A  corresponding  to  the  impulse  response  of  a  transfer  function. 

The  convolution  yields 


on  oo 

0{x,y)=  j  j  oo(a,  /3)y(x  -  a,  y  -  I3)da 


With  the  substitution  —a  =  a  and  —0  =  /?  we  obtain 


OO  OO 

0(x,y)  =  II  ao(-a  ,-0')y{x->fa\y-\-p')da' 


If  a(x,  y)  as  the  spatial  function  to  the  spectrum  A{(t)x,Uy)  is  real  for  real  x  and  y,  then  a  well  known 
theorem  states  that  A{ux,Uy)  has  the  inverse  transform  ao(x,y)  =  a(-x,  — y).  Thus  equation  (14) 
yields  now  again  with  variables  a  and  0 


OO  OO 

0(x,y)  =  II  <»(«.  /J)y(*  +  a.  y  +  0)doi 


This  is  the  crosscorielation  between  a(x,  y)  and  y(x,  y).  The  result  in  equation  (15)  can  be  phiMed 
as  follows:  The  4-f  system  in  figure  6  performs  a  crosscorrelation  between  y(x,  y)  and  a(x,  y)  if  y(z,  y) 
represents  the  input  field  and  the  conjugate  complex  of  the  spectrum  of  a(x,  y)  is  used  in  the  first  foc^ 
plane  of  the  second  lens.  However,  the  output  0(x,  y)  is  the  mirror  image  of  the  desired  crosscorrelation 
0'(x,y)  =  0(-x,-y). 

SLM’s  exhibit  a  discretisation  of  the  x-y  plane  into  pixels  Xi,yj  which  for  briefty  are  denoted  by 
t,  jf.  The  correlation  integral  in  equation  (15)  bec^'mes  the  double  sum 
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oo  oo 

0{i,j)=  a(«./3)y(t  +  a,j  +/3)  (16) 

a=  — OO  ^=  —  00 

or  with  denotations  closer  to  equation  (1) 


0{i,j) 


t  t 


E  E  afc.i  Vi+kj+i 

k=-T  l=-r 


(17) 


where  the  finite  extension  r  of  the  summing  has  been  taken  into  account.  This  limitation  is  possible 
because  ak,i  0  only  for  k  =  — r,  — r  +  1,  •  •  • ,  r  —  1,  r  and  1  =  — r,  — r  -f  1,  •  •  ■ ,  r  —  1,  r.  The  system 
in  equation  (17)  has  (2r  —  1)*  free  parameters.  The  operation  in  equations  (15),  (16)  and  (17)  are 
called  space  invariant  because  the  output  O  depends  on  the  difference  rather  than  on  the  actual  value 
of  the  arguments  of  a(z,  y)  and  y(x,  y).  The  correlation  equation  (17)  realizes  the  desired  correlations 
in  equation  (1)  by  massive  and  fast  parallel  processing. 

The  crosscorrelator  has  a  physical  length  of  4/,  therefore  it  is  usually  designated  as  a  ”4- /-system”. 
It  is  possible  to  reduce  the  physical  length  of  the  optical  correlator  to  about  1.4/  if  the  simple  lenses 
are  replaced  by  lens  systems  [8]. 

4.3  The  optical  addition 

The  three  terms  in  equation  (1)  have  to  be  added.  In  strong  contrast  to  electronics  adders  are  hard 
to  achieve  in  optics.  One  could  add  light  intensities  with  photosensitive  resistors  which  decrease 
its  resistance  with  increasing  light  intensity.  However,  this  effect  is  highly  nonlinear;  in  addition  the 
characteristics  of  the  resistors  fluctuate  widely  £com  sample  to  sample  which  makes  high  precision  signal 
processing  virtually  impossible.  Therefore  we  implement  additions  by  making  use  of  the  additions 
associated  with  the  crosscorrelation  in  equation  (17)  by  putting  all  three  terms  in  equation  (1)  into 
one  single  matrix  [9]  [10].  We  combine  the  inputs  to  the  templates  in  figure  1  into  one  variable  Pi,j(n) 
in  equation  (18)  as  further  detailed  in  figure  7. 


y</j,i/3(n) 

«</2.(r+i)/2 

h 


i  even,  j  even 
i  even,  j  odd 
i  odd 


(18) 


t 

J  1 

2 

3 

4 

1 

Jo 

lo 

lo 

lo 

2 

“1.1 

yi.i 

Wl,3 

yi.2 

3 

lo 

lo 

lo 

lo 

. . . 

4 

W3,l 

yr.i 

“2,2 

y2,2 

1 

t 

1 

* 

Figure  7:  Combined  Input  coding  scheme  for  pij 
Similarly  we  combine  the  A-  and  B-  templates  and  the  bias  dij,  which  is  used  to  shift  the  nonlin¬ 
earity  /,  according  to  equation  (19)  and  figure  8 


=  I 


ak/3,i/j  k  even,  1  even 

fc*/3,(i+i)/2  k  even,  1  odd 

•^*+1/3,1  k  odd 

The  bias  function  Ik,i  is  chosen  according  to: 


(19) 


3r 


X;  E  =  (20) 

le=— r  l=— 3r— 1 

tk,i  is  the  kernel  of  the  optical  correlator.  The  optically  computed  correlation  Cij{n)  between  pij 
and  contains  the  state  variable  Xij{n  1)  as  the  elements  with  even  t, /.  Thus 

3t  3t  t  t 

C3i,3>(n)  =  Zij(n-|-1)=  ^  E  h,lP2i+k,2j+t{^)  =  E  E  <3fc.3tP3i+3t.3j>3t(»)  + 

fc=-3r-l /=-3f-l  4  =  -rJ=-r 

r  T  r  3t 


*=-rl=-r 


k  =  —T  I=-3t-1 


(21) 


A  look  at  the  meaning  of  the  indices  of  tij  and  pij-  in  equations  (19)  and  (18)  reveals 


51 


E.  Lueder,  N.  Fruehauf 


8 


Figure  8:  Combined  Template  coding  scheme 


T  T  T  T  T  3t 

a!i,i(n+l)=  ^  <^k.iVi+k,j+i(n)  +  ^  X]  +  X]  X) 

k  =  -rl=-T  k  =  ~Tl=-T  t  =  -rl=-2r-l 


Figure  9:  Hologram  for  a  corner  detector  (enlarged) 


The  detection  of  the  correlation  result  C]«,2y(n)  poses  another  problem  because  all  sensors  (i.e. 
CCD  cameras  or  photoresistors)  ate  sensitive  only  to  the  intensity  of  the  impinging  light  wave.  As  the 
intensity  is  proportional  to  the  squared  magnitude  of  the  complex  light  wave  amplitude,  it  is  impossible 
to  distinguish  between  positive  and  negative  values  of  C2i,3^(n).  This  ambiguity  can  be  removed  if  a 
constant  D  =  |  minc2i,2j(n)|  =  |minzi,^(n  +  1)]  is  added  before  the  sensing.  In  the  actual  optical 
realization  this  will  be  done  by  adding  D  to  the  rijht  side  of  equation  20: 

r  2r 

X;  X  h.i  ■  lo  =  di,j  +  D  (23) 

*  =  -rJ=-2r-l 

The  increase  of  the  state  variable  Zij(n  +  1)  by  a  constant  D  may  be  canceled  with  a  shift  of  the 
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nonbneai  output  function  /(x)  in  equation  2: 

/(x.,_,(n  +  1))  = /(xi,y(n  + 1)  +  D)  where  /(x)  = /(x  -  Z?)  (24) 

The  nonlinearity  will  be  described  in  the  next  paragraph. 

4.4  The  optical  nonlinearity 

The  nonlinearity  is  realised  as  outlined  in  paragraph  3  by  the  nonlinear  transmission  of  the  SLM.  The 
optical  system  representing  a  CNN  is  shown  in  figure  6. 

4.5  The  optical  storage 

Storage  for  photons  is  not  so  readily  feasible  as  storing  of  electrons  on  a  capacitor.  However,  new  LC- 
materiais  permit  to  overcome  this  shortcoming  of  optical  systems.  Ferroelectric  LC-materials  exhibit 
two  stable  positions  of  its  molecules,  one  in  which  the  cell  is  blocked  and  a  second  one  in  which  it  is 
transparent.  The  molecules  assume  these  positions  with  an  appropriate  addressing  impuls  and  stay 
there  also  after  the  electronic  addressing  has  been  removed  [11].  This  effect  allows  for  the  storage  of 
images  or  of  a  bit-pattern  as  required  in  digital  signal  processing. 


a) 


b) 


c) 


d) 


Figure  10:  Corner  Detection  of  four  squares 
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4.6  The  interconnections 

The  most  attractive  feature  of  optical  processing  is  the  fact  that  the  interconnections  between  the 
neurons  are  provided  inherently  and  in  a  massive  parallel  form  by  the  Fourier  transform  of  the  lens. 
Thus  the  problems  encountered  with  interconnecting  metallization  layers  on  a  chip  just  do  not  exist 
in  optical  processing.  The  weights  of  the  interconnections  are  choosen  according  to  the  combined 
template  ikj  in  equation  (19).  The  conjugate  spectrum  of  j  in  the  filter  plane  of  the  crosscorrelator 
can  be  realized  by  an  SLM  or  as  in  the  following  experiment  by  a  computer  generated  hologram. 
The  hologram  is  a  binary  amplitude  filter  which  is  based  on  a  Lohmann  coding  scheme  with  circular 
overflow  [12]  [13].  In  this  coding  scheme,  the  area  and  the  location  of  small  holes  in  an  opaque  screen 
are  choosen  according  to  the  m<ignitude  and  the  phase  of  the  required  complex  valued  filter.  Figure  9 
depicts  a  filter  which  realizes  the  combined  template  for  a  corner  detector.  An  example  of  an  optical 
CNN  which  performs  corner  detection  will  be  given  in  the  following  paragraph. 

5  Operation  of  the  optical  CNN 

Figure  10  shows  an  example  of  a  CNN  which  performs  a  corner  detection  of  four  squares.  The  original 
input  picture  is  depicted  in  figure  10  a).  Figures  10  b)  -  10  d)  show  the  combined  output  in  time  slot 
1,  2  and  4  during  iteration  thereby  confirming  the  correct  operation  of  the  optical  CNN. 
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Abstract 

By  dcfiitin^  several  notions  of  robustness  for  an  attractor  network,  we  are  able  to  augment  previous 
results  about  the  AdaTron  algorithm  by  explicit  values  for  the  robustness  of  the  optimal  weights.  We 
show,  that  the  symmetry  of  a  problem  is  reflected  by  the  invariance  of  the  optimal  weights.  This 
enables  us  to  deduce  that  a  convex  corner  detection,  using  a  discrete-time  Cellular  Neural  Network 
(DTCNN),  can  not  be  accomplished  vdth  just  one  clock  cycle,  and  we  propose  an  improved  convex 
corner  detector. 


1  Introduction 

It  was  shown,  that  the  AdaTron  ( Adaptative  Percepivon)  algorithm  is  able  to  find  the  most  robust  weights 
of  a  perceptron,  just  taking  actual  constraints  into  account  [2,  3,  4, 11].  Algorithms  solving  the  perception 
problem  [13,  10]  can  be  applied  to  the  learning  problem  of  attractor  networks,  provided  one  is  able  to 
specify  the  trajectories  explicitly.  Hence  we  were  able  to  specify  an  algorithm  which  yields  the  optimal 
weights  for  a  rotationally  invariant  (or  is  utropic  [14])  r-ncighborhood  DTCNN  [5,  12]  on  a  square  grid. 
By  rotationally  invariant  we  mean  that  any  rotation  of  k  •  90°  with  k  &  N  around  the  center  of  the 
template  and  any  reflection  along  the  diagonals  through  the  center,  leaves  the  template  invariant.  In  the 
first  section  we  augment  these  results  by  providing  explicit  formulas  for  the  insensitivity  of  the  optimal 
solution  and  define  several  notions  of  robustness  for  an  attractor  network.  Further  we  will  show,  if  a 
task  is  invariant  with  respect  to  some  group,  so  are  the  optimal  weights.  As  a  side  effect  it  can  happen, 
that  the  optimal  weights  are  invariant  with  respect  to  a  larger  group  of  transformations.  This  leads  to 
the  observation  that  the  conventional  convex  corner  detector  must  fail  in  some  situations.  Using  this 
information  as  a  guide  we  propose  an  improved  one. 


2  Robustness  of  attractor  networks 

Let  U3  consider  discrete  time  neural  networks  [9,  1,  8]  i.e. 

x(J  -f  1)  =  sgn(Ax(i)  +  Bu-fi)  ;  x(0),i,u  6  [-1,1]”  ;  x{l)  6  {-1, 1}"  ioi  I  e  N  ■,  A,B  €  . 

The  design  problem  is  given  by  triplets  of  patterns  where  is  the  desired  output  (after  one 

time  step),  is  the  previous  output,  and  or*'  is  the  input  pattern  for  =  1,  •  •  • , P.  Using  this  notation 
the  design  problem  takes  the  form  -h  b^o’*'  -}-  U)  >  0  for  each  pattern  index  u  and  each  neuron 

k.  a][  resp.  b]^  is  the  l^th  row  vector  of  the  matrix  A  resp.  B.  Note  that  any  solution  can  be  scaled 
by  a  positive  factor  yielding  another  solution  of  the  design  problem.  It  is  reasonable  to  define  the  most 
insensitive  solution  (also  called  optimal  stability  [7])  by  the  following  min-max  problem: 

max  min  Pt{a\C  +  biff"  +  I'k)  subject  to  ||  a*  [j*  +  [j  b*  j|’  -f =  1  for  ib  =  1,  •  •  • ,  n  . 
a..b*,i*  " 

We  will  provide  a  more  rigorous  justification  later  when  we  introduce  the  notion  of  robustness  as  used  in 
statistical  design  by  norm  body  inscription  [15,  11].  Note  that  the  constraint  imposed  on  the  "weights” 
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is,  due  to  our  previous  remarks,  natural  and  moreover  the  problem  would  be  ill-defined  otherwise.  It  can 
be  shown  [7,  2,  4],  that  this  min-max  problem  is  equivalent  to  a  convex  quadratic  programming  problem, 
namely 

niin  (11  a*  |1^  -1-  H  b*  +il)  subject  to  +  u)  >  1  Vi/  and  for  A  =  1,  ■  •  • ,  n  , 

which  can  be  solved  efBciently  for  example  by  the  AdaTron  algorithm. 

It  is  often  necessary  to  impose  constraints  on  the  "weights”  A,B  and  i,  for  example  requiring  sym¬ 
metric  matrices  or  the  translation  invariance  of  the  weights  which  yields  a  DTCNN  [8].  This  can  be  done 
by  specifying  a  linear  parametrisation  S  :  .R"*  — *  jp»xn  "weights”.  We  will  proof  that  even  in 

this  case  the  equivalence  of  the  min-max  problem  and  a  suitably  chosen  convex  quadratic  programming 
problem  still  holds,  hence  providing  am  efficient  tool  for  the  design  of  the  most  insensitive  DTCNN. 

2.1  The  AdaTron  Theorem 

In  the  last  section  we  claimed  the  equivalence  of  a  certain  min-max  problem  to  a  quadratic  optimization 
problem.  More  precisely  it  was  proven  [7,  2,  4]: 

Theorem  1  If  the  perceptron  problem  is  solvable,  i.e.  if  there  is  a  weight  vector  w  6  R",  such  that  for 
all  patterns  €  R”  (v  =  !,••  •  ,P),  w'fl"  >  0,  then  the  following  is  equivalent: 

•  Finding  the  perceptron  of  optimal  insensitivity,  i.e.  the  solution  of 

max  min  w*^*'  subject  to  J1  w  |P=  1  .  (1) 

W  v 

•  Solving  the  convex  quadratic  programming  problem  given  by 

min  11  w  Ip  subject  to  >  1  /orv  =  1,  •  •  • ,  P  .  (2) 

The  solutions  of  (1)  and  w*  of  (2)  are  related  by  =  Aw*  for  an  appropriate  A  >  0. 

Given  w*  the  solution  of  (2)  the  factor  A  in  the  theorem  is  necessary  to  ensiue  the  normalization  of 
w^,  resp.  given  w*,  A  is  used  to  ensure,  that  for  at  least  one  pattern  the  equality  =  1  holds. 

Note  that  since  the  solution  of  a  convex  quadratic  programnung  problem  is  unique,  also  the  perceptron 
of  optimal  insensitivity  is  uniqiicly  defined.  This  theorem  can  be  easily  applied  to  discrete  time  neural 
networks  if  one  sets  for  each  neuron  k  w*  :=  (aj^.b^.i*)  and  0"*  :=  and  solves  the 

quadratic  programming  problem  for  each  neuron.  As  already  noted  it  is  useful  to  consider  problems 
where  not  all  the  coefficients  of  the  weight  vector  w  are  actually  independent  degrees  of  freedom,  for 
example  one  may  require  that  the  "templates”  and  b^  are  rotationally  invariant,  as  it  is  sensible  for 
some  image  processing  task.  Let  S  :  R"'  — *  R"  be  a  linear  parametrization  of  the  weight  vector  w,  i.e. 
each  w  can  be  written  as  w  =  Sv  for  some  vector  v  €  R"'-  Such  a  parametrization  is  not  unique  and  any 
non-singular  transformation  Q  :  R"*  — ►  R"*  defines  a  equivalent  parametrization  S'  :=  SQ.  Using  this 
degree  of  freedom  the  original  problem  (1\  together  with  the  constraint  imposed  by  the  parametrization 
S,  can  be  written  as 

maxminv*^*'  subject  to  v*Q*S*SQv  =  1 , 

V  ¥ 

with  C*'  :=  If  we  choose  Q  such  that  Q*S‘SQ  =  !«,  where  is  the  m  dimensional  iden¬ 

tity  matrix,  we  can  apply  the  original  theorem  and  obtain  a  convex  quadratic  programming  problem. 
Since  Q  and  Im  nrt  non-singular,  this  can  only  be  achieved  if  S*S  is  non-singular.  In  this  case  there 
exists  a  orthogonal  trrmsformation  U  such  that  U^S*SU  =  D  is  diagonal  with  entries  Dii  >  0  and  the 
transformation  Q  is  given  by  Q  =  We  therefore  obtain  the  following  theorem. 

Theorem  2  Let  S  :  R”  — ►  be  a  linear  map  such  that  S*S  is  non-singular.  Ifihe  perceptron  problem 
with  linear  constraints  is  solvable,  i.e.  there  is  a  weight  vector  w  £  R",  such  that  for  all  patterns  8"  £  R” 
(v  =  1,  •  ■  • ,  R^,  vr*6‘'  >  0,  and  3v  £  R"*  such  that  w  =  Sv,  then  the  following  is  equivalent: 

9  Finding  the  perceptron  of  optimal  insensitivity,  i.e.  the  solution  of 

n^minw*^*'  subject  to  |{  w  ||’=  1  and  3v  £  R"'  :  w  =  Sv  . 
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•  Solving  the  convex  quadratic  programming  problem  given  by 

min  II  V  11^  subject  to  v*C  >  1  for  v  =  1,  •  •  • ,  P  . 

where  :=  and  U  is  an  orthogonal  matrix  such  that  U*(S*S)U  =  D  is  a  diagonal 

matrix. 

If  v*  is  a  solution  of  the  second  problem  the  solution  of  the  initial  problem  w*  is  given  by  vr"  =  A  • 
SUD~°  ®v*  for  an  appropriate  A  >  0.  The  value  of  the  optimal  insensitivity  max^min^  =  A. 

The  last  assertion  is  easily  checked  using  the  properties  of  the  transformations  U  and  D.  The  assumption 
that  S‘S  is  non-singular  just  reflects  the  fact  that  the  parametrization  S  may  not  be  redundant.  Since  if 
S‘S  were  singular,  there  would  be  a  parameter  vector  v  0  such  that  (Sv)*(Sv)  =  0.  But  this  is  only 
possible  if  Sv  =  0  and  therefore  there  would  be  two  different  parameter  vectors  (v  and  0)  yielding  the 
same  weight  vector. 


2.2  Robustness  of  attractor  networks 


In  the  last  subsection  we  extended  the  AdaTron  theorem  and  showed  how  it  is  applied  to  discrete  time 
neural  networks  if  no  constrains  are  present.  We  introduced  for  each  neuron  site  j  a  2n  1  dimensional 
weight  vector  w.  Let  us  now  introduce  the  weight  matrix  W  6  R"  x{2n+i)^  where  the  j-th  row  vector 
(written  as  a  column  vector)  is  the  previously  introduced  vector  w  for  the  neuron  at  site  j,  from  now 
on  named  Wj.  Let  us  further  define  the  vector  w  €  by  w*  :=  (wb"-,wi).  A  general  linear 

parametrization  of  a  discrete  time  neural  network  is  now  given  by  a  linear  map  S  :  ii”*  — ►  and 

we  require  that  there  is  a  parameter  vector  v  €  such  that  w  =  Sv.  The  parametrization  S  can  be 
used  for  example  to  express  the  translational  invariance  of  the  weights  or  the  symmetry  of  the  A  and 
B  matrix.  Using  this  notation  the  learning  problem  is  given  by  a  pair  of  patterns  (p",  fl"),  where  p"  €  iT* 
and  5*'  €  and  we  are  looking  for  weights  W  such  that  sgnCWfl")  =  p^.  This  in  turn  is  equivalent 

to  PjyvjO''  >  0,  which  we  will  call  the  attractor  network  problem,  in  analogy  to  the  perceptron  problem. 
With  this  notation  at  hand  let  us  now  relate  the  notion  of  insensitivity  to  the  notion  of  robustness  as 
lued  in  statistical  design  by  norm-body  inscription  [6,  15].  The  robustness  can  be  defined  for  arbitrary 
norms  on  vector  spaces.  We  will  give  the  definition  only  for  the  euclidean  norm,  since  it  is  the  only  one 
we  will  consider.  For  a  more  thorough  treatment  of  this  concept  as  applied  to  neural  networks,  we  refer 
the  reader  to  [16]. 

Definition:  For  any  solution  W  of  the  attractor  network  problem  {p‘',0'')  with  linear  constraints  S 

•  the  relative  robustness  in  pattern  space  r},(W)  is  defined  as  the  solution  of  the  following  optimization 
problem 

max  r  subject  to  VAfl"  :  ||A0‘'||  =  :’||®‘'ll  implies  wjpj(0‘'  -1-  Aff')  >  0  . 

•  the  absolute  robustness  in  pattern  space  iZp(W)  is  defined  as  the  solution  of  the  following  optimization 
problem 

max  R  subject  to  VAfl**  :  ||Afl‘'||  =  R  implies  wjp^(d‘'  -|-  A0‘')  >  0  . 

The  relative  robustness  in  weight  space  ru(W)  is  defined  correspondingly. 


Lemma  1  The  relative  robustness  in  pattern  space  and  weight  space  are  identical  and  can  be  calculated 
as  follows 


Correspondingly  for  Rp  follows 


=  mm  — * — fc — - 


(3) 


iZ-CW)  =  min 

^  •'f  II 


Proof:  Let  us  pick  any  pattern  and  0"  and  any  neuron  j  with  the  associated  weight  vector  Wj  and 
compute  the  maximal  robustness  just  taking  this  pattern  and  neuron  into  account.  From  the  geometrical 
representation  in  Fig.  1  it  is  obvious,  that 


p;wj(0‘'  4-  Afl" )  =  0  and  p;(w,-  +  Aw^)‘«‘'  =  0 


(4) 


57 


Figure  1:  Representatioc  oT  the  hyperplaue  defined  by  vfj  in  the  space  of  patterns  d 

a)  Maximum  norm  pertubation  of  a  pattern 

b)  Maximum  norm  pertubation  of  a  weight  vector 
still  guaranteeing  correct  operation. 

holds.  For  the  perturbation  vectors  in  Fig.  la)  and  b)  we  have 


’■'ll*" II  ■  '="''=-jf^-'-ii*>' 


From  (S)  and  (4)  it  is  easy  to  show,  that 


,  - 

•O  li  nil 


w,-  nil  e- 


PjAw)9‘' 


Therefore  the  maximal  robustness,  taking  all  patterns  into  account,  is  nothing  else,  than  t^dring  the  min- 
imum  over  |[wy||j|s.>)|  for  all  v  and  j  and  this  completes  the  proof  of  (3).  D 

Remark:  Depending  on  the  specific  application,  either  of  these  notions  of  robustness  can  be  the  appro¬ 
priate  one.  In  DTCNNs  with  their  translationally  invariant  templates  all  weight  vectors  Wj  are  equ^  in 
length  and  we  can  apply  the  AdaTron  theorem  and  corresponding  algorithm,  which  can  2u:commodate 
only  one  constraint  (||  W;  ||=:  1)  for  designing  an  attractor  network,  which  is  maximally  robust.  Moreover, 
if  the  patterns  have  equal  length  (e.g.  binary  patterns)  such  a  network  will  also  have  ma-viTniim  absolute 
robustness  in  pattern  space  Rp.  Accounting  for  linear  constraints  can  be  done  as  described  in  section 
2.1  by  applying  the  AdaTron  fJgorithm  to  the  parameter  vector  v  and  the  transformed  patterns  C*'-  The 
maximal  robustness  thereby  obt^ed  in  parameter  and  transformed  pattern  space,  will  result  in  maximal 
robustness  in  the  original  spaces  provided,  that  the  transformation  S  is  norm  preserving. 


3  Convex  corner  detection 

3.1  Group  invariant  learning 

It  is  often  argued,  that  for  rotationally  invariant  (image  processing)  tasks,  rotationally  invariant  templates 
are  the  most  favorable  ones.  The  following  lemma  provides  a  justification. 

Lemma  2  Let  {^i,  *  ■  ■ ,  ^r}  an  orthogonal  (linear)  representation  of  a  finite  group  G.  Let  the  weights 
W  be  the  solution  of 

n^min  W*(j^4‘')  >  0  subject  to  =  I  (7) 

then  the  weights  W  are  invariant  with  respect  to  this  group,  i.e.  g^W  sW  for  all  transformations  f. 
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Figure  3;  I'raining  patterns 

convex  corners  (Fig.  2).  Observe,  that  the  convex  corners  are  characterized  by  the  fact  that  the  pixel  of 
the  input  pattern  and  the  initial  pattern  are  black  and  that  the  cell  in  the  initial  pattern  has  at  least 
one  v/hite  vertical  or  horizontal  neighboring  cell.  Finally  one  has  to  ensure,  that  the  final  output  is  a 
fixpoint.  Therefore,  in  a  third  step,  we  tried  to  learn  the  output  ,  together  with  the  original  input,  to  be 
a  fixpoint. 

For  our  experiments  we  used  a  16  x  16  square  grid  and  we  refer  to  indices  lower  than  1  and  higher 
than  16  as  border  pattern.  We  tried  to  learn  the  three  processing  steps  with  the  initial  pattern  all  white 
and  all  black  and  with  the  initial  pattern  equal  to  the  input  pattern  and  its  inverse.  In  all  cases  it  was 
not  possible  to  learn  all  three  steps.  Though  it  is  possible  with  an  all  white  initial  state  to  learn  the  first 
couple  processing  steps.  The  template  values  are  given  in  Tab.  3,  and  can  be  used  to  perform  a  convex 
corner  detection  using  a  DTCNN  which  has  to  be  stopped  after  two  clock  cycles,  since  otherwise  it  would 
produce  an  all  white  output. 

We  therefore  designed  templates  for  each  one  of  the  two  processing  step  with  the  learning  patterns  of 
Fig.  2.  Again  we  ensured,  that  the  output  is  a  stable  fixpoint.  The  template  values  are  given  in  Tab.  1,  2 
for  border  pattern  values  equal  to  zero  and  -1  (white)  and  for  the  initial  state  equal  to  the  input  and  a  all 
white  state  (for  the  first  processing  step  only).  Actually  the  patterns  in  Fig.  3  were  just  the  first  ones.  In 
order  to  obtain  templates  that  are  not  only  optimal  with  respect  to  the  training  patterns,  but  also  to  the 
task  itself  we  have  employed  the  following  procedure.  We  started  out  with  one  training  pattern  and  kept 
adding  new  patterns  while  monitoring  the  optimal  template  values.  Since  the  optimal  values  changed 
by  less  than  0.1  percent,  after  3  patterns  were  presented  we  are  confident  that  the  template  values  are 
optimal  with  respect  to  the  task. 


4  Conclusion 

It  was  shown,  that  the  AdaTron  cilgorithm  yields  the  optimal  weights  for  various  notions  of  robustness 
even  in  the  presence  of  symmetries,  just  taking  actual  constraints  into  account.  From  an  engineering  point 
of  view,  the  actual  value  of  the  robustness  is  extremely  important  and  we  had  to  augment  our  previous 
results.  The  experiments  we  performed  for  the  improved  convex  corner  detector  were  quite  promising, 
since  the  algorithm  is  efficient  (in  our  case  within  minutes  on  a  Sun  workstation)  and  the  value  of  the 
robustness  is  fairly  high.  Still  the  main  drawback  of  any  design  algorithm  is,  that  if  the  problem  is 
not  solvable  with  just  one  processing  step  a  trajectory  has  to  be  designed.  For  problems  possessing  a 
symmetry,  we  showed  how  "non-geometrical”  transformations  can  hinder  a  solution  in  a  single  processing 
step.  In  turn  this  information  can  be  used  as  a  guide  for  the  design  of  an  appropriate  trajectory. 
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Figure  2:  Patterns  indistinguishable  for  an  isotropic  DTCNN 

Proof;  The  proof  is  indirect.  Suppose,  that  W  is  the  solution  of  (7)  and  let  us  define  the  weight  vector 
w  as  w  :=  1/r  g'^'W  cind  compute  its  length. 

<w,w>=l5;^<S«W,5PW><l^||sW||||3PM^||  =  l  ,  (8) 

Pi?  Pi? 

since  the  transformations  are  orthogonal.  Hence  there  is  a  A  >  1  such  that  l|Aw|j  =  1  and  moreover,  by 
construction,  AW  is  invariant  with  respect  to  the  group  G.  For  any  transformation  g^  and  pattern  we 

A  A 

(Aw)‘(/4‘')  =  -  52  WWg^C  =  -y\  W^g^C  >  A  min  W*g‘C  •  (9) 

r  ^  T  ^  Vi 

Since  A  >  1  and  the  solution  of  (7)  is  unique,  we  have  an  contradiction,  which  proves  the  claim.  □ 
The  above  ideas  can  also  be  applied  to  attr^u:to^  networks.  The  neuron  at  site  i  of  an  attractor  network 
performs  the  mapping  pj'  =  sgn(W7^‘').  If  we  require,  that  the  patterns  gP^‘'  are  mapped  in  the  same 
way,  the  above  considerations  show,  that  the  optimal  weights  W,-  are  invariant  with  respect  to  the  group 
G.  Actually  the  group  G  could  be  different  for  each  neuron,  yielding  different  constraints  for  the  weights. 
Thb  shows  that  in  this  context  we  require  a  local  invariance.  As  we  wi}l  see  for  the  convex  corner  detection 
it  can  happen  that  there  is  a  larger  group  of  transformation  {/i^,  •  •  • ,  h^}  which  contains  the  group  of 
transformations  as  a  subgroup,  but  also  leaves  the  weights  Wi  invariant.  It  is  then  also 

impossible  to  distinguish  between  the  patterns  and  the  patterns  .  Applying  these  considerations 
to  a  DTCNN  it  is  easily  checked  that  a  isotropic  (im^e  processing)  task  implies  that  the  optimal  templates 
are  isotropic  ones.  The  coefficients  of  the  a  rmd  b  template  hence  satiesfy  a,t  =  Opq  iff  the  sets  {jsj,  |t|} 
and  {|p|,  1^1}  are  equal.  Let  us  now  illxistrate  the  consequences  of  the  above  mentioned  "larger  group” 
for  the  learning  problem.  To  this  end  we  define  the  vector  U  =  {Ui,  •  •  •,  U^)  =  (oio,  Ooi,  o-ioi  oo-i)-  A 
rotation  about  90°  is  given  by  a  cyclic  permutation  of  the  elements  of  U.  A  refection  along  the  diagonal 
permutes  Ux  with  U2  and  with  U\.  Hence  the  invariance  of  the  template  implies  that  all  4  coefficients 
are  equal.  Such  a  vector  is  now  invariant  with  respect  to  any  permutation  of  its  elements,  including  the 
"non-geometrical”  permutation  of  Ux  with  Cfa.  Therefore  the  patterns  in  Fig.  2  can  not  be  distinguished. 

3.2  Improved  convex  corner  detection 

Following  the  procedure  outlined  at  the  end  of  section  2.2,  we  derived  an  algorithm  which  yields  the  most 
robust  weights  for  a  rotationally  invariant  r-neighborhood  DTCNN  on  a  (M  x  iV)  square  grid  [llj.  For 
the  design  of  the  templates  we  used  this  algorithm  and  by  combining  the  assertion  about  the  optimal 
insensitivity  (theorem  2)  with  the  definitions  and  explicit  expressions  for  the  robustness  of  attractor 
networks  we  are  able  to  provide  explicit  values  for  the  optimal  robustness.  The  considerations  at  the  end 
of  the  last  subsection  showed,  that  the  detection  of  convex  corners  can  not  be  accomplished  using  a  single 
processing  step  of  a  DTCNN  (Fig.  2).  To  overcome  this  difficulty  we  designed  a  two  step  process  for  the 
convex  corner  detection  of  a  binary  image.  Since  straight  lines  can  not  be  distinguished  from  corners 
(Fig.  2),  we  use  the  first  processing  step  to  mark  those  boundary  points  of  the  image  which  appear  to  be 
part  of  a  straight  line  by  placing  an  additional  black  pixel  right  next  to  it.  (gray  pixels  in  Fig.  3).  More 
precisely  whenever  a  pixel  is  white  and  the  pattern  inside  a  1-neighborhood  "window”  centered  at  this 
pixel  appears  to  be  part  of  a  striught  fine  crossing  this  window,  we  change  the  white  center  pixel  into  a 
black  one.  This  output  is  employed  as  the  initial  state  for  the  next  processing  step  which  extracts  the 
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Table  1:  Optimal  template  values  for  the  first  processing  step,  for  the  initial  pattern  equal 
to  the  input  and  an  all  white  pattern. 


Table  2:  Optimal  template  values  for  Table  3:  Optimal  template  values  for 


the  second  processing  step  (b.p.v  =  bor-  the  stopped  process  (b.p.v  =  border 

der  pattern  value).  pattern  value). 
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We  use  Bayesian  methods  to  design  cellular  neural  networks  for  signal  processing  tasks  and  the 
Boltzmann  Machine  learning  rule  for  parameter  estimation.  The  learning  rule  can  be  used  for 
models  with  “hidden”  units,  or  for  completely  unsupervised  learning.  The  latter  is  exemplified  by 
unsupervised  adaptation  of  an  image  segmentation  cellular  network,  in  particular  we  apply  the 
learning  rule  to  adaptive  segmentation  of  satellite  imagery. 


1  Introduction 


The  Bayesian  or  Maximum  Posterior  approach  is  a  very  successful  device  for  signal  processing 
[15,  7,  11],  a  particular  attraction  is  that  it  leads  to  algorithms  that  map  well  onto  networks  of 
locally  connected,  simple  processing  elements  ie.  cellular  neural  networks  [4].  With  the  advent  of 
low-cost  massively  parallel  hardware,  this  virtue  may  end  up  being  decisive  for  (near)  future  real 
life  applications.  However,  while  the  Bayesian  approach  allows  formulation  of  collective  models 
that  solve  complex  signal  processing  tasks,  general  and  flexible  tools  for  parameter  estimation 
are  lacking.  Direct  maximum-likelihood  estimation  is  hampered  by  the  difflculty  of  obtaining 
analytical  expressions  for  derivatives  of  normalization  constants  [1].  Besag  has  introduced  two 
methods  based  on  approximate  maximum-likelihood  estimation,  the  coding  method  [1],  and  the 
pseudo-likelihood  method  [2].  Both  methods  pose  difficulties  if  the  image  model  includes  non¬ 
observable  attributes  ie.  hidden  units.  In  this  contribution  we  discuss  the  use  of  the  Boltzmann 
Machine  learning  rule  [10]  for  parameter  estimation.  The  learning  rule  may  be  applied  to  general 
situations  with  hidden  units  without  coixiplication.  We  address  in  this  presentation  unsupervised 
learning. 

Since  the  phase-space  distribution  of  a  dynamical  system  in  contact  with  a  heat-bath  is  a  Gibbs 
distribution;  sampling  from  such  distributions  have  been  central  to  simulations  of  statistical 
physics  systems.  The  standard  simulation  tool  is  due  to  Metropolis  et  al.  [13].  Geman  and 
Geman  [7]  introduced  Metropolis  sampling  from  Gibbs  distributions  as  a  simulation  tool  for 
visual  reconstruction  and  showed  that  a  Simulated  Annealing  strategy  [12]  could  improve  the 
speed  of  the  sampling  process.  The  sampling  process  implements  a  stochastic  neural  network 
with  symmetric  connections.  Hinton  and  Sejnowski  [10]  studied  supervised  learning  in  such 
networks  and  introduced  the  term  Boltzmann  Machines  to  emphasize  the  relation  to  statistical 
physics.  Invoking  the  Mean  Field  approximation,  Peterson  and  Anderson  derived  a  deterministic 
selfconsistent  set  of  equations  for  the  time-averages  of  the  dynamical  variables.  Using  these 
averages  in  the  learning  rule,  they  obtained  substancial  improvements  in  speed  and  performance 
[14]. 
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In  the  next  section  the  Bayesian  approach  to  signal  processing  is  outlined,  and  we  discuss  the 
design  of  cellular  networks  using  the  Mean  Field  approach.  In  section  three  we  show  how 
the  Boltzmann  Machine  learning  rule  may  be  applied  for  parameter  adaptation.  Section  four 
contains  experiments  on  image  segmentation  and  concluding  remarks. 


2  Bayesian  Signal  Processing 


The  Bayes  approach  to  signal  processing  has  a  long  tradition,  see  e.g.  [11,  7],  or  [5]  for  a 
recent  introduction.  The  basic  idea  is  to  consider  both  the  source  (un-degraded)  signal  and 
the  degradation  as  stochastic  processes.  The  Bayes  formula  can  then  be  used  to  construct  the 
distribution  of  the  reconstructed  signal  (x),  conditioned  on  the  observed  degraded  signal  (y): 


P{Ay) 


Piy\x)P{x) 

P{y) 


(1) 


According  to  the  standard  interpretation  the  conditional  distribution  is  the  product  of  the 
distribution  of  the  degradation  process:  P{y\x)  =  P(x  — *•  y),  and  the  prior  distribution  of  the 
reconstructed  signal  P(x).  P(x|y)  of  equation  (1)  is  referred  to  as  the  postenor  distribution.  A 
useful  estimate  of  the  recontructed  signal  is  given  by  the  location  of  the  mode  of  the  posterior 
distribution,  the  socailed  Maximum  A  Posteori  (MAP)  estimate.  In  the  following  we  derive  the 
posterior  distributions  for  an  image  segmentation  model. 


2.1  Image  Segmentation 

Segmentation  is  an  important  step  in  many  computer  vision  systems,  however  even  in  its  simplest 
form:  binarization  of  a  grey-scale  image  there  exist  no  established  standard  solution!.  Here  we 
use  the  Bayes  scheme  to  derive  a  simple  cost-function  that  can  be  minimized  by  a  cellular  neural 
network.  The  resulting  cost-function  is  identical  to  the  one  used  by  Camevali  et  al.  [3].  The 
target  signal  is  a  smooth  binarization  of  a  grey-scale  image  dj,  in  terms  of  two- valued  pixels 
Sj  €  {— 1,-fl}.  The  prior  distribution  is  designed  to  emphasize  smoothness: 

/  N  N 

=  exp  I  -  5])  53 

V 

defines  the  connectivity,  hence  the  unit  cell  of  the  cellular  network.  Here  we  just  use 
the  nearest  neighbors.  More  complex  connectivity  structures  have  been  designed  for  modelling 
textural  features  [6]. 

We  assume  the  signal  degradation  to  consist  in  addition  of  white  Gaussian  noise.  This  degra¬ 
dation  process  leads  to  the  following  conditional  distribution: 


fXSj  -  5i.)M 


(2) 


We  want  to  approach  real  data  for  which  the  noise  variance  is  unknown,  hence  this  parameter 
has  to  estimated  as  part  of  the  learning  process.  As  above  we  use  Bayes  to  combine  and  obtain 
the  posterior  distribution.  Clearly  it  is  of  the  Gibbs  form^,  with  a  cost-function  given  by  the 
negative  logarithm  of  the  posterior  distribution:  —  logP[5|d].  We  note  that  the  state  dependent 
part  of  the  cost-function  is  linear  in  the  parameters  M(j,f)  and  wj  = 


2.2  Network  Design 

The  Mean  Field  annealing  method  for  estimation  of  averages  over  Gibbs  distributions  is  well 
documented  in  the  litterature  see  e.g.  Hertz  et  at.  [9].  The  cellular  neural  network  is  designed 
to  minimize  the  Mean  Field  free  energy  and  this  can  be  done  either  in  analog  mode: 

=  -{Sj)  +  tanh  M(i, ^(5;.)  +  Wid\  j  (4) 

or  in  discrete  time  mode: 

(Sj+')  =  (1  -  ^')(Sj)  +  ^  tanh  L‘  /  f;  mums}.)  +  j  j  (5) 

where  the  time-scale  A/r  can  be  used  to  regularize  the  stability  of  the  iteration  process  in 
digital  implementation[14].  0*  is  quantify  the  annealing  schedule,  in  this  work  we  use  the  simple 
schedule:  /3‘  =  -f  -  0^)- 

In  summary,  the  unit  cell  of  the  cellular  network  contains  one  unit  approximating  the  local 
thermodynamic  average:  (5j),  and  one  input  unit  dj.  We  assume  that  connects  nearest 

neighbors  symmetrically  with  weight  Wn,  and  the  weights  to  the  input  units  are  all  Wd. 


3  Boltzmann  Machine  Learning 


In  order  to  apply  the  cellular  network  above  we  have  to  estimate  the  parameters  (denoted 
w  =  (wn,Wd))‘  Since  our  network  is  based  on  the  Gibbs  distribution  we  invoke  the  Boltzmann 
Machine  learning  rule  [10,  14,  9].  The  objective  of  the  Boltzmann  Machine  learning  nde  is  to 
minimize  the  Kullback  information  distance  between  a  target  distribution  (of  which  the 

training  set  is  a  finite  sample),  and  the  distribution  Pu,[5|d]  sampled  by  the  current  (stochastic) 
network  with  parameters  w. 

The  learning  rule  is  formulated  for  a  general  system  specified  in  terms  of  inputs,  hiddens,  and 
outputs:  (x,  h,  y).  Since  the  states  of  the  hidden  units  are  unknown  for  the  learning  examples,  we 
compare  the  marginal  distributions,  i.e.  the  distributions  integrated  over  the  hidden  variables: 

*A  distribation  of  the  form  P{x)  =  Z~* exp(—E(x)fT),  where  E[x)  is  a  cost-function,  bounded  from  below, 
and  T  is  a  parameter 
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Figure  1:  Subsampled,  128x128  pixel,  preprocessed,  Landsat  image  providing  the  raw  (thresh- 
olded)  evidence  for  water  in  the  Igaliko  region  of  Greenland  (left),  and  the  output  evidence 
obtained  by  the  unsupervised  cellular  neural  network  (right) 

=  /  Px  /  Z),  /  [X.  A,  ,1  log 

/  Dx  is  short  for  fUj  The  learning  algorithm  is  derived  by  gradient  descent  minimization 
of  the  information  distance.  The  recursive  learning  algorithm  reads: 


-  tn"  =  »/  ((n„  (x,  h,  y))}clamped  “  (x,  h,  y))/ree)  (7) 

for  a  any  cost-function  linear  in  the  parameters  E(x,h,y)  =  (^>h,y)wv,  where 

fl„  (x,h,y)  is  an  expression  in  the  stochastic  variables.  {...)ciamped  indicates  that  the  average  is 
performed  with  respect  to  the  current  Gibbs  distribution,  with  fixed  values  for  all  the  stochastic 
variables  that  are  specified  as  input  or  output  variables  in  an  example  of  the  database,  i.e.  (z,  y). 
Similarly  (...)/,ee  is  the  average  with  only  input  variables  fixed.  In  brief  we  can  characterize  the 
learning  process  as  follows:  the  parameters  are  adjusted  to  minimize  the  difference  between  the 
correlations  in  situations  with  and  without  the  teacher  specifying  the  correct  output  [10,  9]. 

We  can  employ  the  above  formalism  for  unsupervised  learning  if  we  let  our  output  units  S  play 
the  role  of  hidden  units  and  partition  the  input  image  d  into  a  Boltzmann  “input”  part  and 
a  Boltzmann  “output”  part.  The  Boltzmann  learning  process  then  adapts  the  model  until  we 
reach  a  parameter  set  for  which  the  “output”  part  of  the  image  is  estimated  correctly  from 
“input”  part.  Our  procedure  can  be  viewed  as  an  example  of  statistical  cross-validation. 
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Figure  2:  Unsupervised  Boltzmann  learning  of  the  noise  parameter  of  the  cellular  segmentation 
network. 

4  Experimental  and  concluding  remarks 

Our  case-study  concerns  the  segmention  of  a  subsampled  Landsat  satellite  image.  The  input 
signal  is  i  preprocessed  128  *  128  pixel  Image  representing  the  evidence  for  water  in  the  Igaliko 
region  ir  Greenland.  The  preprocessing  scheme  establish  the  evidence  using  four  frequency 
bands.  Fart  of  the  image  has  been  classified  manually  into  five  classes  and  the  evidence  is  the 
result  of  a  simple  linear  model  relating  the  intensities  in  the  four  bands  to  the  classification. 
Thresholding  the  evidence  at  zero  results  in  Figure  la).  We  adapt  the  noise  parameter  of  the 
cellular  etwork  in  unsupervised  mode  using  a  gradient  descent  parameter  of  0.15.  In  Figure 
lb)  we  Si.ow  the  segmentation  of  the  adapted  network.  The  convergence  of  the  noise-parameter 
Wd  towai  ds  the  selfconsistent  optimal  value  is  presented  in  Figure  2. 

In  concl'u  jion  we  have  shown  that  the  Boltzmann  Machine  learning  rule  can  be  used  for  identifi¬ 
cation  of  parameters  in  cellular  neural  networks  designed  using  Bayesian  reasoning.  By  invoking 
a  crossvaiidation-like  procedure  we  were  able  to  adapt  the  parameters  of  the  cellular  network 
without  supervision,  hence  generalizing  our  earlier  results  on  parameter  estimation  in  cellular 
networks  with  hidden  units  [8]. 
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Abstract 

This  paper  presents  a  learning  algorithm  similoi  to  the  Backpropagation-Through-Time  approach. 
The  algorithm  is  based  on  the  minimization  of  an  error  criterion,  which  is  defined  as  the  product  of 
a  function  of  the  state  at  a  given  time  and  the  integral  of  an  entire  time  function  of  the  state  over  the 
trajectory  prior  to  this  time.  The  technique  of  the  calculus-of-variation  will  be  used  to  evaluate  the 
gradient  of  the  error  in  the  parameter  space,  which  can  be  used  to  descend  to  a  minimum  on  the  error 
surface.  We  will  adopt  this  theory  to  CN'Ns  and  show  some  simple  examples  of  the  learning  of  CNN 
parameters. 


1  Introduction 

We  concentrate  on  a  subclass  of  neural  networks:  Cellular  Neural  Network;  (CNNs)  [2,  7],  where  the 
interconnections  between  the  neurons  or  cells  are  translationally  invariant  and  only  local.  Therefore,  the 
number  of  weights,  which  have  to  be  learned  or  designed,  is  very  small.  Because  of  the  piecewise  linear 
transfer  function  of  the  CNNs  the  gradient  algorithms  of  Almeida  [1]  and  Pineda  [9, 10, 11],  which  are 
backpropagation  algorithms  generalized  for  recurrent  neural  networks,  can  not  be  used.  Based  on  the 
idea  of  Pearlmutter  [8]  to  design  the  trajectory  of  recurrent  dynamic  neural  networks,  we  will  use  the 
trajectory  of  the  CNN  state  to  gain  information  about  the  gradient  of  a  modiried  error  functional. 

Other  approaches  towards  the  systematic  design  of  CNNs  have  been  proposed  by  Zou  et.  al.  [16],  and 
later  by  Slot  and  Kacprzak  [IS].  Chua  and  Thiran  [3]  provide  a  method  for  synthesizing  CNNs  for  simple 
applications,  and  Seiler  et.  al.  [14]  show  how  to  systematically  design  a  CNN  with  stable  and  unstable 
outputs  while  simultaneously  maximizing  its  robustness.  Except  for  [3],  which  is  restricted  to  relatively 
simple  problems,  the  correct  operation  of  the  network  is  not  guaranteed  since  only  the  stable  or  unstable 
patterns  of  the  dynamical  system  are  designed. 

In  the  next  section  we  will  use  the  calculus-of-variations  technique  to  minimize  an  error  functional  of 
a  dynamical  system  in  general.  The  derived  equations  will  be  applied  to  CNNs  in  Section  3,  and  some 
examples  will  be  shown  in  Section  4. 

2  The  Backpropagation-through-Time  Algorithm 

The  Backpropagation-through-Time  (BTT)  [8, 6]  algorithm  can  be  derived  from  solving  a  classical  vari¬ 
ation  problem  for  any  dynamical  system  [13],  which  is  given  by  a  system  of  differential  equations 
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(1) 


X  =  F(x,^p), 

where  x  e  IR”  is  the  system  state  and  p  a  parameter  vector  of  the  system. 

The  performance  of  the  system  is  assumed  to  be  measured  by  a  scalar  functional  E  which  bookkeeps 
the  errors  over  the  whole  transient,  i.e.  over  the  finite  interval  [0,  T]: 

T 

E  =  LlixiT))jL2ix,t)dt.  (2) 

0 

Li  ;  ]R"  -♦  IR"  measures  the  error  at  the  given  time  T,  whereas  Lj  :  IR"  x  IR  — ►  K."  is  integrated  over 
the  given  interval,  to  take  into  account  the  effect  of  a  change  of  the  parameters  on  the  trajectory.  This 
can  be  used  to  learn  a  specific  trajectory  as  well. 

The  problem  is  to  minimize  the  error  functional  (2)  under  the  equality  constraint  given  by  (1).  Using 
the  vector  equivalent  of  the  Lagrange  multiplier  A  6  IR"  the  solution  to  the  constrained  problem  can  be 
obtained  by  minimizing 

T 

E'  =  y[Ll(x(T))L2(x,0  +  A'^(F(x,<,p)-x)]df.  (3) 

0 

A  necessary  condition  for  an  extremum  of  E'  is  that  the  first  variation  of  E'  be  zero:^ 


T 

6E'  =  J[lI^Sx^  +Ll(x(T))^^x+A^(g^x-a)  +  (F(x,f,p)-xfdd<.  (4) 

Since  6E’  must  equal  zero  independent  of  the  first  variation  Sx,  (4)  yields  the  following  equations: 

The  Euler-Lagrange  equation: 

The  associated  transversality  condition: 

.  (6) 

0  i=T 

The  gradient  can  be  obtained  by  the  differentiation  of  (2)  with  respect  to  p: 

With  the  partial  derivative  of  (1)  with  respect  to  p: 

—  -  ^ 

Sp  dx  dp^  dp' 

and  using  the  definition  of  A  fi-om  (5)  together  with  (6),  (7)  can  be  simplified  to 

BE  /  /'dF\  . 


'The  partial  derivative  of  a  scalar  function  /(x)  with  respect  to  a  vector  %  is  defined  to  be  a  row  vectw 
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A  necessary  condition  for  an  extremum  of  £  is  =  0. 

Minima  of  the  error  functional  can  be  found  using  various  gradient  techniques  like  conjugate  gradient 
methods  or  Newton-like  methods.  The  BTT-algorithm  works  as  follows: 

begin  Minimizing  the  Error  Functional  E 
start  with  a  random  parameter  vector  p 

repeat 

integrate  x  from  t  =  0  to  T 

integrate  A  from  t  =  T  to  0  and  simultaneously  evaluate  the  gradient 
perform  a  gradient  step  to  reduce  the  value  of  the  error 
until  gradient  of  E  becomes  zero 

end. 


3  The  BTT-Algorithm  for  CNNs 


The  backpropagation-through-time  algorithm  can  be  applied  to  CNNs  [2,  7],  and  the  CNN  equations 
(10)  and  (11)  have  to  be  used  for  (1): 

+»■  +r 

^c,d  ~  ~^c,d  "t"  ^  ^  f^yUc+i/fd+ti  "i"  ^^^^c+l/,d+^i\  "I"  ^  5  (10) 

i/=— r  ii=:—r 

Vc.dW  =  f(XcA^))  =  5  (|1  +  -  U  -  a:c,<i(0l)  •  dD 

The  state  variables  Xc,d,  where  1  <  c  <  and  I  <  d  <  M  define  the  two-dimensional  grid,  can  be 
arranged  in  a  state  vector  x  €  IR”,  with  n  =  NM.  The  summations  are  restricted  to  the  valid  range 
of  each  index.  The  parameter  vector  p  of  a  CNN  is  composed  of  the  variables  of  the  cloning  template 
a{u,  p),  b(u,  p)  and  i,  where  -r  <  i/  <  +r  and  -r  <  p  <  and  r  is  the  neighborhood. 


The  partial  derivative  of  the  state  equations  with  .respect  to  the  state  variables  can  be  computed  from  the 
right  hand  side  of  (10): 


OFcJrv 

dx 


ro,  if|i/l: 

-1  -I-  o(0,C)/'(xc.a) ,  if  = 
'=-‘'  [  a(-i/,  -p)r{xc,d) ,  else . 


>  r  or  \p\  >  r ; 
=  0  and  p  =  0; 


The  partial  differentiation  of  (10)  with  respect  to  the  parameters  yields: 


OF, 


c,d  _ 

—  “f^c+v.d+ii  > 


dF 


c,d  _ 


=  1  . 


dbiu,p)  ~  ’  di 

With  the  use  of  A,  which  has  the  same  dimensions  as  x,  the  gradient  of  the  error  becomes: 


(12) 


(13) 


dE 

J  N  M 

~  I  fi^c+t/,d+n)^c,ddt  1 

0  c=l<i=l 

(14) 

da{v,p) 

dE 

db{v,p) 

"K  N  M 

—  I  tic+i/.d+u^c.ddt  , 

0  c=l<<=l 

(15) 

dE 

di 

?  N  M 

=  J'LH^c.ddl. 

n  c=U=i 

(16) 

If  the  vector  valued  functions  Li  and  Lz  are  choosen  to  be  the  squared  errors  of  the  corresponding  cells 

5  (/(*c.d(r)) 

\  2  1  2 
“  J/c.d)  ^^c,d  ~  2  (■^(^^■‘^(d)  ~  yc,d'^  » 

(17) 

where  y*  ^  is  the  desired  output  of  the  network  for  cell  c,  d,  the  error  functional  becomes 

^  =  i;  E  5  [IMT))  -  yufj  5  (/(x.,i(*))  -  . 


c=l  d=l 

Therefore  (5)  and  (6)  become: 


+r  +r  -  , 


1/—  —  T  /!=”’* 


1 

X^T)  =  {/(xcdT))  -  y;,,)  fMT))  j  \  (fixat))  -  yufdt . 


4  Examples 

The  examples  in  this  section  use  linear  extended  CNNs,  which  are  trained  to  do  different  tasks  using 
the  previous  algorithm.  In  the  case  of  several  learning  samples,  the  error  and  gradients  are  computed 
separately  and  then  added  for  the  whole  sample.  Throughout  the  following  examples  the  initial  parameter 
was  set  to  zero:  p  =  0,  the  time  limit  was  T  =  10  and  the  minimization  of  the  error  functional  was  done 
with  the  conjugate  gradient  minimization  method  of  Fletcher  and  Reeves  [4].  We  will  show  the  resulting 
template  values,  and  as  a  measure  of  the  computational  effort  of  the  minimization  the  number  of  function 
evaluations  (running  of  the  CNN  forward  and  integrating  the  error  over  the  transient),  and  the  number 
of  gradient  evaluations  (running  backward  in  time  while  simultaneously  computing  the  gradients). 


4.1  State-based  Logical  OR 

The  design  of  a  simple  two-cell  CNN  which  produces  an  output  of  both  cells  which  is  equal  the  logical 
"OR"  of  the  initial  state,  where  -fl  is  interpreted  as  ’true’  and  —1  as  ’false’  was  reported  in  [14].  As 
there  is  no  input  of  the  CNN  the  operation  is  called  state-based.  The  four  different  combinations  of 
logical  values  were  used  as  the  learning  samples.  After  three  gradient  steps  with  a  total  number  of  seven 
forward  and  backward,  integrations  the  error  was  zero.  The  resulting  template  values  and  the  phase- 
plane  trajectory  of  the  four  different  initial  conditions  of  this  two-cell  CNN  are  shown  together  with  the 
boundary  of  the  basins  of  attraction  in  Fig.  4.1.  Although  the  self-feedback  is  slightly  smaller  than  one, 
this  template  obtains  a  stable  CNN.  The  symmetry  of  the  templates  is  obtained  automatically  because  of 
the  symmetry  of  the  problem. 


a=  0.716  0.964  0.716 
b  =  0.000  0.000  0.000 


1=  0.033 


Fig.  4.1:  The  phase-plane  trajectories  together  with  the 
boundary  of  the  basins  of  attraction  and  the  template  val¬ 
ues  of  the  state-based  logical  OR  CNN 


42  Connccted-Component-Detector 


To  learn  the  task  of  "Connected-Component-Detection"  [5]  we  used  all  possible  combinations  of  black 
and  white  cells  in  a  linear  CNN  with  five  cells,  which  were  the  32  learning  samples.  After  only  13 
gradient  descents  with  47  forward  and  backward  integrations  the  error  and  the  gradient  were  zero.  In 
fig.  (4.2)  the  learned  template  values  for  this  problem  are  given. 


0.430 

1.509 

-0.430 

0.000 

0.000 

0.000 

-0.003 

Fig.  4.2:  The  resulting  template  values  of  the  Con¬ 
nected  Component  Detector 


4.3  Two-Cell  Oscillator 

The  last  example  will  show  that  this  algorithm  can  be  used  to  train  one  cell  of  a  simple  two-ceU  CNN 
to  follow  a  given  trajectory.  The  desired  trajectory  of  the  output  of  cell  1  was  a  trapezoidal  function  of 
the  time  with  an  amplitude  of  1,  a  period  of  S  units  of  time  and  an  absolute  value  of  the  slopes  of  2.  For 
this  example,  the  functions  Li  and  hi  were  chosen  differently  to  measure  only  the  error  of  cell  1  over 
the  trajectory,  so  that  the  error  functional  E  was 

T 

E^\j  {Kxm-ymfdt.  (21) 

0 

Fig.  (4.3)  shows  the  trained  output  of  cell  1,  starting  from  the  initial  state  x  =  (0, 0.7)’’’  together  with 
the  corresponding  values  of  the  template. 


-0.672 

1.197 

4.000 

0.000 

0.000 

0.000 

0.006 

Fig.  4.3:  The  output  of  cell  1  of  the  two-cell  osdl- 
lator  and  the  corresponding  template  values 


5  Conclusion 

Our  goal  has  been  to  find  a  learning  algorithm  for  CNNs,  which  is  able  to  learn  the  parameters  of  a 
CNN  to  perform  a  transition  from  an  initial  state  to  a  state,  whieh  yields  a  prescribed  output  We  have 
shown  a  gradient  technique  which  is  able  to  learn  the  parameters  such  that  a  given  error  functional  is 
minimized.  Of  course  we  find  not  always  the  global  minimum,  because  the  proposed  algorithm  uses  the 
gradient  of  the  error  functional  in  the  parameter  space.  Thus  the  algorithm  stops,  if  the  gradient  is  zero, 
whether  this  is  a  local  minimum  or  a  global  one.  Another  drawback  of  the  algorithm  is  the  computational 
complexity.  In  order  to  compute  the  gradient  of  the  error  functional  not  only  the  differential  equation 
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of  the  CNN  has  to  be  integrated  forward  in  time,  but  also  the  Euler-Lagrange  equation,  which  is  of  the 
same  dimensionality  has  to  be  integrated  backward  in  time.  Therefore  the  complete  trajectory  of  the 
states  has  to  be  stored.  This  restricts  the  application  of  this  algorithm  to  small  examples.  But  in  most 
cases  this  is  not  a  severe  restriction,  as  the  learning  can  be  achieved  with  relatively  small  samples. 
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Abstract 

It  has  been  shown  that  the  supervised  learning  of  the  steady-state  outputs  in 
a  generalized  Cellular  Neural  Network  (CNN)  is,  in  general,  equivalent  to  a  kind 
of  constrained  optimization  problem.  The  objective  function,  also  be  called  as  the 
error  function,  is  a  measure  of  the  distance  between  the  sets  of  desired  steady-state 
outputs  and  actual  ones.  The  constraints  are  due  to  a  set  of  design  requirements 
which  have  to  be  met  for  providing  the  qualitative  and  quantitative  properties  for 
the  networlc  such  as  bipolarity  of  the  steady-state  outputs,  complete  stability  etc. 

The  approach  presented  in  the  paper  uses  the  idea  of  the  penalty  function  method 
in  optimization  theory  where  the  constrained  optimization  problem  is  transformed 
into  an  unconstrained  one  by  adding  to  the  error  function  the  terms  corresponding 
to  the  constraints.  A  gradient  descent  algorithm  has  been  proposed  for  solving  the 
resulting  unconstrained  optimization  problem.  The  learning  algorithm  developed 
generalizes  the  recurrent  backpropagation  algorithm  in  [6]  into  the  generalized  CNN 
which  is  a  rather  general  class  of  dynamical  neural  networks  including  CNN,  multi¬ 
layer  Perception,  continuous  Hopfield  network  as  special  cases. 

1  Completely  Stable  Generalized  CNNs 

The  generalized  CNN  considered  in  this  paper  is  a  dynamical  neural  network  recently 
introduced  in  [1],  It  is  indeed  a  generalization  of  CNN  with  respect  to  both  connection 
topology  and  the  circuit  structure  of  the  cells.  As  shown  in  Figure  1,  each  cell  of  a  gener¬ 
alized  C.NN  is  an  n  th  order  dynamical  circuit  which  is  a  casc^adc  of  a  linear  summation 
circuit,  a  linear  dynamical  circuit  and  an  output  nonlinearity.  There  are  two  kinds  of 
connection  weights  in  a  generalized  CNN  ;  i)  m,  ;  weighting  the  outputs  j/j  of  the  cells,  ii) 
2,;  weighting  the  external  inputs  u--.  In  this  paper,  the  learning  is  assumed  to  be  accom- 
pli.shed  through  modification  of  the  connection  weights  only.  For  the  sake  of  generality, 
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we  define  the  input  vector  as  v  =  [v^v'^]'^.  Where,  =  [ui,U2, ,  and  respectively 
Vx  —  [a:^(0),  ...,xf(0)]^  denotes  the  vector  of  external  inputs,  and  respectively  the  vector 
of  initial  states  with  t  is  the  number  of  cells.  The  network  weight  vector  is  defined  as 
W  =  [u^^,  ,  zj, zj]'^.  Where,  Wj  =  [lyj.i,  •••>  ^j,YjV  weight  vector  of  the 

j  th  cell  such  that  its  i  th  element  belongs  to  the  connection  between  the  cell  Cj  and  the 
i  th  neighbor  of  Cj  ;  and  Zj  s  are  defined  similarly. 

For  a  given  input  vector  v  ,  a  generalized  CNN  with  a  chosen  weight  vector  W  will  pro¬ 
duce  an  output  y{t)eR^.  When  the  input  vector  is  held  fixed,  the  output  y{t)  tends  to  a 
constant  vector  y(oo)  ,  called  the  steady-state  output,  if  the  network  is  completely  stable 
[1].  Throughout  the  paper,  we  are  interested  in  completely  stable  generalized  CNNs  only. 
Such  generalized  CNNs  defines  an  algebraic  map  between  the  input  and  the  steady-state 
output  vector  spaces.  The  complete  stability  of  a  network  can  be  ensured  by  a  set  of 
conditions  imposed  on  the  connection  weight  vector  W  [1]  ,  [2]  ,  [5]  ,  [8]  .  It  is  shown 
in  [2]  that  a  CNN  is  completely  stable  if  the  connection  weights  have  the  property  of 
symmetry  i.e.  tn,  ,■  =  luj ,  for  all  i  and  i.  Such  constraints  constitute  one  part  of  the  design 
constraints  we  mentioned  and  they  can  be  described  by  the  following  set  of  nonlinear 
algebraic  equalities  and  inequalities. 

5.(IF)<0  i£{l,2,...,Ci}  (1) 

In  addition  to  the  stability,  some  other  qualitative  or  quantitative  properties  can  be  desired 
for  a  dynamical  neural  network.  As  proved  in  [2],  the  bipolarity  of  the  steady-state  outputs 
can  be  obtained  by  choosing  self-feedback  weights  u;,,,  greater  than  a  constant  A,  .  Such 
constraints  constitute  the  second  part  of  our  design  constraints  and  they  can  be  described 
by  the  set  of  algebraic  equalities  and  inequalities  in  (2). 

k{W)<0  i£{l,2,...,C2}  (2) 


2  Supervised  Learning  as  a  Constrained  Optimiza¬ 
tion  Problem 


The  supervised  learning  of  the  steady-state  outputs  in  a  generalized  CNN  attempts  to 
approximate  an  unknown  input- (steady-state)output  map  d  =  F(v)  by  minimizing  an 
error  function  B(IV).  The  network  is  trained  with  the  following  set  of  pairs  which  are 
samples  of  the  map  d  =  F{v)  '• 

. (A-i")}  (3) 


Where  v*  and  d’  represents  the  input  and  desired  (steady-state)  output  for  the  i  th  sample, 
respectively.  The  error  function  E{W)  to  be  minimized  is  a  measure  of  the  difference  be¬ 
tween  the  desired  and  actual  (steady-state)  output  sets.  E{W)  is  defined  a.s  the  following 
summation  of  the  instantaneous  errors  E'{W)  . 

E(W)  =  ^E'{W)  (4) 

1=1 


Where  E'{W)  =  D(r/‘(oo),<i')  denotes  the  distance  between  the  vectors  y*(oo)  and  (T  in 
a  vector  space  having  a  metric  Z)(y(oo),d).  Now,  the  supervised  learning  of  the  steady- 
state  outputs  in  a  completely  stable  generalized  CNN  can  be  formulated  as  a  constrained 
optimization  problem  expressed  in  (5). 
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mmE{W)  (5) 

Where,  C  =:  {W\gi[W)  <  0  i£{l,2,  and  /i,(W)  <  0  ze{l,  2, C2}}  .  The 

method  preferred  here  for  solving  this  minimization  problem  is  the  well-known  penalty 
function  method.  The  reason  for  using  the  penalty  method  is  the  possibility  of  converting 
the  constrained  optimization  problem  in  (5)  into  an  unconstrained  one  which  can  be 
solved  by  a  gradient  descent  algorithm.  The  stochastic  version  of  the  learning  algorithm 
developed  in  this  paper  reduces  for  unconstrained  cases  to  the  popular  backpropagation 
algorithm  which  is  indeed  a  stochastic  gradient  descent  algorithm.  The  gradient  descent 
method  is  proposed  here  for  minimizing  the  augmented  error  function  E{W)  obtained  by 
adding  to  the  error  function  the  penalty  terms  as  in  (6). 

E{W)  =  E{W)  +  fiPiW)  (6) 

Where  /i  >  0  and  P(W)  is  a  continuously  differentiable  function  of  W  satisfying  the 
conditions  i)  P{W)  >  0  for  all  W  ,  and  ii)  P{W)  =  0  if  and  only  if  WeC  with  C  defined 
in  (5)  .  It  is  known  [13]  that  as  /x  goes  to  infinity  any  minimum  point  of  the  augmented 
error  in  (6)  will  converge  to  a  solution  of  the  constrained  minimization  problem  described 
in  (5). 

The  deterministic  gradient  descent  algorithm  proposed  here  is  defined  by  the  iterative 
algorithm 

W{k  -hi)  =  W{k)  -  6(i-)(Vw^[W(/c)]f  (7) 

Where,  VwE{W)  is  the  gradient  vector  of  EI^W)  with  respect  to  W  ,  and  t{k)  is  a 
nonnegative  scalar  minimizing  —  c(Vvv.^[W(/j)])^]  •  It  is  known  [13]  that  such  an 

algorithm  is  globally  convergent  and  its  limit  points  are  the  minimum  points  of  E(W).  In 
the  learning  procedure  proposed  here,  the  coefficients  are  set  to  a  small  constant  value  t  , 
called  the  learning  rate  coefficient.  One  can  select  e  by  trial  and  error.  For  large  values  of 
e  ,  the  algorithm  may  not  be  convergent.  Small  values  of  e  may  result  in  slow  convergency. 
For  the  Euclidean  metric  and  for  a  chosen  penalty  function,  the  gradient  of  E{W)  can  be 
given  as 

VwE[W{k)]  =  VwE[W{k)]  +  fiVwP[W{k)] 

=  (j/j(°°)  ■“  (ma.T:[0^.(W)])2 

t=l  j=l  i=l 

-c  pVwi:(ma.r[0,/t.(W)])2 

t=i 

=  E  E  2  -  “t)  Vw!/;(oo)  +  /<  Z2maj:[0,},(fV)]  V«.a(Vr) 

i=l  j=l  i-1 

C2 

-1-  27nax[0,/x.(iy)]Vw'/t,(W)  (8) 

i=l 

3  The  Stochastic  Gradient  Descent  Algorithm 

In  the  algorithm  given  above,  the  weights  are  adapted  after  the  calculation  of  the  gradient 
of  the  total  error  function  by  computing  the  contributions  from  ail  of  the  training  samples. 


Such  a  mode  for  learning  is  called  batch  learning.  For  large  sets  of  training  samples, 
the  batch  mode  for  learning  needs  much  computation  to  be  performed  for  every  update 
of  the  weights.  It  might  then  be  preferable  to  have  an  algorithm  which  updates  the 
weights  after  every  presentation  of  a  traning  sample.  This  well  known  idea  of  stochastic 
approximation  leads  us  to  develop  the  stochaistic  version  of  the  algorithm  in  (7).  In 
the  stochastic  algorithm,  the  training  samples  are  viewed  as  random  samples  from  an 
unknown  distribution  and  the  expected  error  function  is  estimated  with  the  instantaneous 
error  function  E'‘{W{k))  .Thus  ,  the  connection  weight  vector  W(k)  is  adapted  according 
to 

W{k  +  1)  =  W{k)  -  e{k)  {VwE'‘[W{k)]f  (9) 

Where  E’^{W)  =  E^{W)  +  ^P{W)  with  P{W)  defined  in  (6).  It  is  known  [12]  that 
stochastic  approximation  guarantees  the  convergence  if  the  learning- rate  coefficients  t{k) 
satisfies  the  conditions  in  (10)  and  (11). 


OO 


Y,e{k)  =  OO 

i=l 

(10) 

OO 

X^c^(fc)  <  OO 
i=l 

(11) 

The  learning-rate  coefficients  t{k)  should  decrease  slowly  in  the  sense  of  the  divergent 
sum  in  (10)  as  well  as  quickly  in  the  sense  of  the  convergent  sum  in  (11). 

4  Conclusion 

The  supervised  learning  algorithm  developed  in  this  paper  can  be  considered  as  a  gen¬ 
eralization  of  the  recurrent  backpropagation  algorithm  [6]  to  the  generalized  CNN  where 
the  connection  weights  can  not  be  chosen  freely  due  to  some  conditions  imposed  on  them. 
The  algorithm  can  be  used  not  only  for  dynamical  neural  networks  such  as  CNN,  contin¬ 
uous  Hopfield  network  but  also  for  algebraic  neural  networks  having  some  constraints  on 
connection  weights.  The  proposed  algorithm  is  useful  for  training  the  CNN  to  perform 
image  processing  tasks. 
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Abstract:  The  learning  process  for  a  Discrete-Time  Cellular  Neural  Network  is  formulated  as 
an  optimisation  problem.  This  involves  nnnimizing  an  objective  function,  which  is  a  measure 
of  the  errors  in  the  desired  input-to-output  image  mapping  process  performed  by  the  network. 
With  this  approach,  the  learning  algorithm  finds  the  trajectories,  so  they  no  longer  have  to 
he  designed  by  the  user. 


1  Introduction 

Discrctc-Timc  Cellular  Neural  Networks  (DTCNN)  were  introduced  in  [i].  The  DTCNN  is 
a  clocked  feedback  network  with  translationally  invariant  weights.  The  transition  from  one 
clock  cycle  to  the  next  is  determined  by  the  equations 


./€:V(c)  d(i.N{c) 


+1  if  x^{k)  >  0 
-1  if  x^\k)  <  0 


where  /;•  is  a  positive  integer  corresponding  to  the  clock  period,  a  —  (o'^)  and  b  =  \jf-)  are 
the  templates  (weig.btcd  connections),  i  is  the  cell  bias.  y{h)  ~  iifilk))  is  the  output  image 
of  the  net  at  time  and  u  =  is  the  input  image.  A’ (c)  is  a  neighborhood  of  ceil  c.  In 

this  paper,  the  inputs  arc  restricted  to  be  binary-valued  and  constant,  i.c.  v.''  £  {—1.  ] ). 

In  most  case;;,  neural  networks  arc  used  to  obtain  some  kind  of  mapping  from  input  data  onto 
output  data.  In  contrast  to  tliis  global  point  of  view,  the  templates  of  CNNs  and  DTCXNs 
arc  usually  derived  from  local  rules  at  cell  level:  either  only  the  fixpoints  of  the  network 
or  a  detailed  trajectory  is  designed  by  the  user  ([2],  [3]  or  [4]).  A  Learning  Algorithm  for 
Supervised  Training  using  a  Noisy  ERror  surface  for  DTCNNs  (LASTNERD)  is  introduced  in 
this  paper.  It  tacldcs  the  problem  of  learning  the  template  coefficients  from  a  different  angle: 
an  error  measure  for  the  mapping  process  of  input  images  onto  the  desired  output  images  is 
minimized.  This  (global)  information  is  the  only  information  which  the  algorithm  uses,  so 
that  it  can  find  a  trajectory  without  the  help  of  the  user,  and  sets  the  templates  accordingly. 


The  objective  function  is  derived  in  the  second  section,  and  some  prominent  features  of 
the  resulting  error  surface  arc  discussed  in  section  3.  Section  4  describes  the  LASTNERD 
algorithm.  Section  5  gives  a  shon  summary  of  the  experiments  which  were  performed  to 
test  the  algorithm,  and  the  last  section  summarizes  the  ideas  of  the  paper  and  presents  a 
short  outlook. 
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2  The  Objective  Function 

The  process  of  learning  is  put  into  the  form  of  an  optimization  problem,  in  which  an  objective 
function  o,  which  depends  on  a  parameter  vector  v  =  (a,b,/)  containing  the  template 
coefficients,  is  minimized. 

The  goal  of  the  learning  process  is  to  find  the  parameter  vector  Vopt  that  guarantees  an  error- 
free  mapping  of  a  number  of  input  images  onto  the  corresponding  desired  output  images.  An 
objective  function  has  to  quantify  the  errors  which  the  network  makes  in  the  mapping  process. 
An  obvious  approach  is  the  following;  an  image  is  fed  into  the  network  as  the  input  image, 
the  initial  state  is  set  to  either  the  input  image,  to  a  white  (—1)  or  to  a  black  (-1-1)  image. 
Then  the  network  runs  until  it  reaches  a  stable  state.  The  Hamming  distance  between  the 
converged  output  image  and  the  desired  output  image  corresponding  to  the  input  image  is 
used  as  the  error  measure. 

Oscillatory  behavior  can  be  a  big  problem  for  DTCNNs,  so  the  case  that  the  network  does  not 
reach  a  fixed  point  also  has  to  be  taken  into  account  The  network  is  constantly  monitored  in 
order  to  detect  cyclic  behavior.  Since  a  DTCNN  is  a  clocked  structure  with  binary  outputs, 
this  can  be  done  in  a  fairly  straightforward  way.  If  oscillations  are  detected,  the  objective 
function  punishes  this  behavior  by  taking  on  the  maximum  possible  value.  The  optimization 
algorithm  is  then  going  to  drive  the  parameter  vector  away  from  this  (unstable)  point  in  the 
parameter  space  towards  regions  with  lower  objective  function  values. 

Let  p  denote  a  pair  of  an  input  image  u  and  the  corresponding  desired  output  image  =  (y^). 
Then  the  modified  Hamming  distance  dmCPjV)  for  a  given  input-ou^ut  image  pair  p  at  a 
certain  point  v  in  the  parameter  space  becomes 

'  ^  2 

A  if  the  net  is  stable 

am(P,Vj=<  c=l  W 

.1  if  the  net  oscillates 

where  0  <  <  1.  y^(u,  v)  is  the  converged  output  of  cell  c,  when  the  net  was  started  with 

the  input  image  u  from  the  input-output  image  pair  p,  and  y^  is  the  desired  output  of  cell  c  in 
the  ouq^ut  image  of  the  input-output  image  pair  p.  M  is  the  number  of  cells  in  the  network. 
In  general,  a  neural  network  is  required  to  learn  not  only  one  but  a  lai:ge  number  n  of  input- 
output  image  pairs  p,.  Let  P  =  {pi,. . .  ,p„}  denote  the  set  of  all  input-ouq>ut  image  pairs 
which  the  network  has  to  learn.  An  ideal  objective  function  should  average  the  modified 
Hamming  distances  dm{pi)  over  all  n  input-output  image  pairs  p,,  but  very  often  this  is  not 
advisable  due  to  the  computational  cost 

To  circumvent  this  problem,  only  a  subset  5  C  of  all  input-output  image  pairs  is  used  in 
the  averaging  process.  This  subset  S  of  size  I  <n  is  selected  randomly.  Let  y  =  {^i, . . . ,  ?« } 
be  a  permutation  of  Then  the  objective  function  o(v)  can  finally  be  written  as 

,  I 

o{y)  =  ■jY^dm{pg.,v)  (3) 

‘  »=i 

Again  we  have  0  <  o(v)  <  1.  It  is  obvious  that  a  multiple  evaluation  of  the  objective 
function  with  v  fixed  will  result  in  slightly  different  values.  This  happens  because  different 
input-output  image  pairs  p,  are  used  for  the  evaluation  of  o(  v)  each  time  the  function  is  called. 


This  random  selection  can  also  be  viewed  as  a  noise  component  in  the  objective  function. 
This  noise  component  has  an  important  property:  it  vanishes,  if  a  parameter  vector  Vopt 
has  been  found,  which  guarantees  a  perfect  input-to-output  mapping  of  the  network.  In  this 
case,  the  modified  Hamming  distance  between  the  desired  and  the  actual  output  will  be  zero, 
no  matter  which  input-output  image  pairs  pi  €  are  selected  during  the  evaluation  of  the 
objective  function. 

The  noise  component  in  the  objective  function  is  crucial  to  the  functioning  of  the  algorithm. 
This  will  become  obvious  later  in  this  paper.  The  amount  of  noise  in  the  objective  function 
can  be  influenced  by  the  number  of  input-output  image  pairs  used  during  the  evaluation  of 
the  objective  function.  A  ratio  of  ^  close  to  one  results  in  a  small  noise  component,  and  a 
ratio  close  to  zero  in  a  strong  noise  component  in  the  objective  function. 


3  The  Error  Surface 


Before  introducing  the  LASTNERD  algorithm  it  makes  sense  to  look  at  some  peculiarities  of 
the  error  surface,  i.e.  the  objective  function  o  as  a  function  of  the  parameter  vector  v. 

It  is  quite  obvious  that  —  due  to  the  nonlinear  (SIGNUM-)  characteristic  of  each  cell,  which 
assures  the  binary  nature  of  the  cell  output  —  the  objective  function  is  only  going  to  change 
at  those  points  in  the  parameter  space  where  x^{k)  in  (1)  is  equal  to  zero. 

Rewriting  this  condition  into  a  more  familiar  form,  we  get 

x^{k)  =  •  e^{k  -  1)  =  0  (4) 

where  v  is  once  again  the  parameter  vector  containing  the  all  the  coefficients  of  the  a-  and 
b-template  and  the  bias  i.  The  vector  —  1)  combines  the  respective  cell  output  values 
y‘^{k  —  1)  and  values  from  the  input  layer  from  the  neighborhood  of  cell  c  at  time  k  —  ], 
and  a  constant  +1  for  the  cell  bias.  Since  the  input  values  are  restricted  to  be  binary-valued 
in  this  paper,  the  vector  e^{k  —  1)  will  only  have  binary  entries,  thus 


r  ±1  for/z  =  l,...,yV- 1 
\  1  for  fi  =  N 


(5) 


where  N  is  the  number  of  inputs  of  a  cell  and  the  dimension  of  v  and  e‘^(Jb  —  1)  (for  an 
r -neighborhood  on  a  square  grid,  N  =  (2r  -f- 1)^  holds).  (4)  describes  a  hyperplane  in  , 
of  which  there  can  be  as  many  as  2'^  due  to  the  constraints  on  the  components  of  —  1 ) 
in  (5). 

Thus  the  error  surface  consists  of  a  large  number  of  convex  polytopes  Ri,  which  arc  bounded 
by  the  hypcrplanes  in  (4).  The  objective  function  will  be  constant  on  these  polytopes.  Let 
dRi  denote  the  hull  of  polytope  Ri.  Hence,  gradients  on  the  error  surface  arc  either  zero  (for 
V  ^  Ri\  dR)  or  undefined  (for  v  6  dR).  In  addition,  ihe  objective  function  is  corrupted  by 
the  noise  component  that  was  explained  in  the  proceeding  section. 
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Fig.  2  A  2-dimensioaal  example 
of  the  Alternating  Variable  method 


Fig.  1  A  2-dimensional  cross-section  through  an  error 
surface  of  a  3-dimensioaal  example  (v’^  =  (a_i,ao,ai)) 


Fig.l  shows  a  two-dimensional  cross-section  through  a  (N  =  3)-dimensional  error  surface. 
The  convex  polytopes  and  the  noise  component  of  the  error  surface  are  clearly  visible.  Due 
to  the  binary  nature  of  the  vector  e‘(A:  —  1),  the  boundaries  dRi  in  a  two-dimensional  cross- 
section  are  always  intersecting  at  right  angles  or  run  in  parallel. 


4  The  LASTNERD  Algorithm 

For  the  reasons  stated  above,  methods  of  optimization  using  gradient  techniques  like  Quasi- 
Newton  or  Steepest  Descent  methods  (see  for  example  [S],  [6])  are  not  applicable. 

Direct  Search  methods  are  an  alternative  to  gradient  search  strategies.  They  are  usually  more 
expensive  in  terms  of  the  number  of  function  evaluations,  but  on  the  other  hand,  they  are 
much  more  flexible. 

The  LASTNERD  algorithm  is  based  on  the  well  known  Alternating  Variable  Method  (as  for 
example  described  in  chapter  2  in  [S]).  In  this  method,  line  searches  in  parallel  to  the  axes 
of  the  coordinate  system  are  performed  repeatedly.  A  line  search  procedure  is  supposed  to 
find  the  minimum  Omin  =  o(Q'nun)  of  o{a)  =  o(vgtart  -I-  Q!v),  where  q  is  a  real  number  and 
V8tart,v  are  iV -dimensional  vectors  (see  section  2.6  in  [6]  for  more  details  on  line  search 
algorithms).  The  result  of  each  line  search  is  then  taken  as  a  new  (temporary)  optimum.  The 
whole  process  is  repeated  until  the  objective  function  falls  below  a  desired  level. 

Fig.2  shows  a  simple  two-dimensional  example.  In  that  figure,  the  thin  solid  lines  are  contour 
lines  of  a  function  defined  over  and  the  thin  dotted  lines  show  the  direction  of  the  line 
searches. 

Due  to  the  special  properties  of  the  objective  function,  some  modifications  to  the  original 
Alternating  Variable  Method  had  to  be  made.  The  LASTNERD  algorithm  works  as  follows: 


begin  LASTNERD 

randomize  the  initial  coefficients; 

repeat 

for  for  all  coordinate  axes  do 

begin 

-  Normalize  the  template  coefficients; 

-  perform  a  line  search  parallel  to  an  axis  of 
the  coordinate  system,  starting  from  the  current 
optimal  point,  and  keep  the  result  as  the  new 
optimum; 

end 

until  template  coefficients  have  been  found 

which  guarantee  a  zero  (or  sufficiently 
low)  objective  function; 
end  LASTNERD 

To  avoid  any  kind  of  cyclic  behavior,  the  line  searches  along  the  different  coordinate  axes 
have  to  be  performed  in  a  random  order. 

The  line  searches  remain  the  biggest  problem  in  the  algorithm.  The  function  0(0)  in  the 
line  search  process  is  a  noisy  piecewise  constant  function,  which  will  be  very  difficult  to 
minimize.  The  way  line  searches  are  performed  in  the  LASTNERD  algorithm  is  by  taking 
samples  of  o(a)  from  an  interval  around  a  =  0.  Finding  the  best  parameters  for  this 
sampling  process  is  relatively  difficult  and  requires  some  experience.  If  the  distance  between 
the  samples  is  too  small,  than  the  algorithm  becomes  too  expensive  from  a  computational 
point  of  view.  If  on  the  other  hand  the  samples  are  too  far  away  from  each  other,  the  line 
search  algorithm  might  jump  over  "trenches"  in  the  objective  function.  The  size  of  the  interval 
around  a  =  0  is  another  problem.  The  LASTNERD  algorithm  uses  a  compromise:  If  the 
objective  function  is  still  high,  a  larger  interval  la  and  a  larger  distance  between  the  samples 
is  chosen.  As  the  objective  function  becomes  smaller,  the  interval  Ja  becomes  smaller  and 
the  sampling  rate  higher. 

Another  critical  issue  is  the  terminating  condition  for  the  algorithm.  Due  to  the  noise  in  the 
objective  function,  a  zero  objective  function  does  not  necessarily  imply  that  the  input-ouq)ut 
mapping  works  for  all  image  pairs.  A  better  idea  is  to  terminate  the  algorithm,  when  a  number 
of  consecutive  evaluations  of  the  error  function  have  returned  the  value  zero,  thus  making 
sure  that  a  large  number  of  input-output  image  pairs  were  tested  during  the  evaluation  of 
the  objective  function. 

Any  nonlinear  optimization  algorithm  is  usually  prone  to  getting  stuck  in  local  minima.  In 
simulations  of  the  LASTNERD  algorithm,  it  turned  out  that  the  noise  component  in  the 
objective  function  is  actually  very  helpful,  since  it  may  help  the  algorithm  to  jump  out  of 
local  minima.  Still,  the  amount  of  noise  (controlled  by  the  ratio  has  to  be  determined 
experimentally,  because  too  much  noise  in  the  objective  function  might  prevent  the  algorithm 
from  descending  to  lower  objective  function  values. 


5  Experimental  Results 

The  algorithm  is  working  properly  for  moderate  numbers  N  of  template  coefficients  (around 
20  to  30).  While  M,  the  number  of  cells  of  the  neural  network,  influences  only  the  execution 
time  of  the  algorithm,  simulations  suggest  that  the  performance  of  the  LASTNERD  algorithm 
deteriorates  with  increasing  N,  which  corresponds  to  the  dimensionality  of  the  underlying 
optimization  problem. 

The  algorithm  was  tested  with  well-known  problems  from  literature,  like  Edge  Detection, 
Connected  Component  Detection,  Hole  Filling,  etc..  For  these  tasks,  templates  are  known 
which  solve  the  respective  problem.  Using  a  10  x  5  network  with  a  translationally  invariant 
1 -neighborhood  in  a-  and  b-templates  and  a  bias  (19  template  coefficients),  the  algorithm 
found  templates  solving  the  given  problems. 

The  algorithm  was  finally  applied  to  the  problem  of  straight  line  (horizontal  and  vertical) 
detection,  for  which  no  known  lemplates  with  a  1 -neighborhood  exist.  The  LASTNERD 
algorithm  was  not  able  to  find  templates  for  an  error-free  operation  of  the  DTCNN,  but  it 
found  a  suboptimal  solution  resulting  in  a  very  low  pixel  error  rate. 

6  Conclusion  and  Outlook 

This  paper  introduces  the  idea  of  mapping  the  learning  task  for  DTCNNs  onto  a  nonlinear 
optimization  problem.  The  method  used  for  solving  this  optimization  problem  —  a  modified 
Alternate  Variable  method  —  is  relatively  flexible,  but  not  very  efficient,  since  it  does  not 
take  into  account  the  specific  structure  of  the  DTCNN. 

Still,  the  fact  that  the  LASTNERD  algorithm  could  find  the  templates  for  the  examples  shows 
that  learning  for  DTCNNs  can  indeed  be  done  in  this  way. 

Currently,  more  advanced  numerical  algorithms,  among  them  genetic  algorithms,  are  applied 
to  the  inherent  optimization  problem.  Using  these  algorithms,  some  interesting  new  applica¬ 
tions  for  DTCNNs  were  found,  which  are  to  be  reported  in  the  near  future. 
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1 .  I n  t  r  oduc  t i on 

It  is  expected  that  neural 
network  system  such  as 
Hopfield  neural  network 
(HF-NN)  may  solve  the 
general  associative  and 
optimum  problems.  However. 
HF-NN' s  need  m^  synapses, 
where  m  is  the  number  of 
neurons  in  the  system.  It  is 
very  difficult  to  fabricate 
such  a  large  HF-NN.  One  of 
the  method  to  solve  the 
problem  is  to  use  Cellar 
Neural  Networks  (CNN)  in 
which  the  number  of  synapses 
is  determined  by  dn/2  where 
d  is  the  number  of  synapses 
incident  to  each  neurons. 
But  there  are  synthesis 
problems  to  determine 

weights  for  association. 

In  this  paper,  we  discuss  a 
new  synthesis  method  of  CNN. 

2.  BN-mapping  decision 

We  assume  that  the 
structure  of  CNN  is  fixed. 
In  this  section,  we  discuss 
the  mapping  from  each  bit  of 
the  data  to  a  neuron.  We 


call  it  a  BN-mapping. 

We  assume  that  fin  is  one 
of  the  n  fundamental 
memories  and  that  each 
fundamental  memory  Iik=  [Aik 
•"  Amx]  consists  of  m 
elements,  each  of  which  is  1 
or  -1. 

The  energy  function  of  full 
connection  type  NN  is 
defined  by  the  following 
equa  t i on [ 3 ] 

E  (») --ae'Faj/2.  (D 

where  F  is  the  connection 
matrix  which  is  defined  as 

F  =  S(IIkIIh*  —  Em),  (2) 

k~  1 

where  Em  is  m  dimensional 
unit  matrix  and  the  super 
subscript  t  represents  the 
transport  of  a  matrix.  When 
the  f u 1 1 -connec t i on  type  NN 
recall  j-th  fundamental 
memory,  a  energy  Ej  is 
presented  by 

Ej--hj^Fhj/2.  (3) 

In  the  CNN  which  has  local 
connection,  the  connection 
matrix  S(F)  can  be 
represented  by  a  sparse 
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matrix.  Though,  F  is 
replaced  S (F)  which  is 
sparse,  we  must  select  an 
optimal  function  S  <  (.  ) 
from  the  set  of  {  S  i  (.  ) 
li  =  l,  •••  }.  The  decision 

depend  on  the  BN-mapping. 
So,  we  must  determine  the 
BN-mapping  so  that  the 
associated  energy  function 
of  CNN  can  be  decreasing  as 
possible  as  we  can.  Because 
there  are  some  fundamental 
memories,  we  must  solve 
min-max  problem  such  that 
the  largest  energy  function 
can  be  minimized.  That  is 
the  max-min  problem 

represented  by 
u  =  min(max(-hj^Si  (F)  llj/2 
I  j  =  1,  n  )  I  i «1, n) . 

Examp  1  e 

The  CNN  which  is  7X5  array 
structure  stores  the  12 
fundamental  memories  shown 
in  fig.  2.  The  connection  of 
each  neuron  is  defined  in 
fig.  1.  By  the  proposed 
method,  the  BN-mapping  is 
given  as  fig.  3. 

3.  Weight  decision 
3. 1  Convergent  radius 

We  modify  the  connection 
matrix  Si  (F)  to  the 
following  connection  matrix 
for  CNN; 

N=SAk(S  (hnllK’^-Em)  )  ,(4) 

k=l 

where 

Ak-diagfaiK,  •”  ,  amn)  •  (5) 


Now,  we  assume  that  CNN  can 


memo  r i z  e 

a 

f  und amen  t  a  1 

memo  r  y  ll  i  . 

When 

the  output 

of  CNN  is 

h,  , 

a  i  npu  t  o  f 

the  j-th. 

neuron  JVj  is 

dec i ded  by 

Iji  — 

(6) 

whe  re  d j  is 

a  threshold  of  a 

neuron  Nj, 

then 

hj  1  =  s  i  gn 

(  7  1  I  )  , 

and  t  j  is 

j-th 

r  ow  e 1 emen  t 

vec  t  or  of 

ma  t  r  i  X 

N.  If  the 

prove  vector  or 

the  output 

of  HF-NN  i 

s  h  I 

with  r  bit 

errors,  the  input  of  j-th 

neuron  /  j i ’ 

1  s 

decided  by 

/ji-2rs^Iji’  ^/ji+2rs,  (7) 

where  s  is  the  biggest  value 
in  the  absolute  of  the 
elements  of  matrix  N. 

It  is  important  that  if  the 
right  side  and  the  left  side 
of  equation  (7)  are  equal  in 
sign,  CNN  always  correct  r 


bit  errors 

It  means 

that  r 

is  convergent  radius 

of  CNN 

o  r  sma 1 1 e  r 

than  it. 

In  this 

paper,  we 

decide 

such  that 

s>l,  and  we  must 

determine 

{ A 1 ,  dll 

i  =  l,  m) 

so  that 

r  equals  to  rmox 

which  is 

the  maximum 

value 

of  r. 

Note  that  we  can  decide  s. 
Because  sign(.  )  cares  only 
the  sign  of  its  parameter. 
3.2  Computing  method  of  the 
we i gh  t 

By  modifying  equation  (7) 
and  using  s-1,  the  following 
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equations  are  obtained 
/ji-2r^0  I  Aji>0, 

/jj+2r^0  I  hji<0.  (8) 

And  they  are  modified  to 
Aj  ,  (/j  ,-2r)  ^0.  (9) 

Condition  (9)  is  applied  to 
all  fundamental  memories  and 
all  neurons.  /j  i  is 


descr ibed 

as  foil owi ng 

equa  t ion 

I J  i“  (f)ajK(Aj-kllK*“Cj)  )  lli 
k“  1 

+  dj. 

(10) 

So,  Condition  (9)  can  be 

modified  to 
r 

Aji  {2ajK(Aj>tllK^“Cj))  111 

ir=  1 

+  dj)-2r^0.  I  i-l,  Ji,  j-1,  m 

(11) 

where  Cj  is  j-th  row 
element  of  Em*  We  can  get 

the  following  conditions 
using  5-1  and  equation  (6) 
n 

SsjuAjhAi  K“1^0f 
it- 1 

I  i,  7  =  1.  m,  j  (12) 

1  “  jKAjKAik^Of 

1  i,  7-1.  m,  i^j  (13) 
We  have  to  get  rmax  on 

condition  (11),  (12)  and 

(13)  for  learning.  Because 

these  equations  are 

described  by  linear 

equations,  we  can  use  the 

linear  mathematical 

programming,  for  instance 

Symplex  method,  to  solve  it. 

3.  3  Exampl  e 

By  the  proposed  method, 
the  weights  are  given  as 


fig.  4.  We  have  simulated 
the  associative  operation  of 
CNN  by  using  the  resulted 
weights.  Fig.  5  shows  the 
some  output  pattern  with 
weights  of  the  equation  (2) 
and  proposed  weights.  The 
CNN  by  the  proposed 
weighting  method  recalls  all 
fundamental  memories,  but 
the  CNN  by  the  using  the 
weight  determined  the 

equation  (2)  can  not  recall 
i  t. 
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Fig.  2  Fundamental  memories. 
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Fig.  4  The  weights  of  CNN. 
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Abstract  -  Among  Artificial  Neural  Networks,  CNN’s  have  experienced  an  unprecedentiy 
known  explosion  of  researches  and  applications,  due  to  their  unique  property  of  using  only 
local  connections.  So  their  use  has  become  attractive  for  many  purposes.  We  will  just  recall: 

1.  Modeling  the  physiology  of  the  brain. 

2.  Modeling  the  processes  of  learning  and  classifying  by  human  beings  and  animals. 

3.  Modeling  the  behavior  of  large  physical  systems,  usually  studied  by  means  of  statistical 
mechanics. 

4.  Using  them  as  a  new  paradigm  for  very  powerful  electronic  computers,  well  suitable  to 
be  implemented  using  VLSI. 

As  engineers,  we  are  mainly  interested  in  the  last  item,  but  it  is  important  to  not  forget 
the  strong  links  that  exist  between  this  item  4)  and  the  preceding  ones.  Considering  such 
links  could  make  easier  finding  solutions  to  the  many  questions  still  open  in  the  systematic 
design  of  CNN’s. 

The  purpose  of  this  talk  would  be  starting  a  fruitful  discussion  on  the  answers  to  the 
following  three  questions: 

1.  What  CNN’s  are?  (i.e.  definition) 

2.  How  do  CNN’s  operate?  (i.e.  behavior) 

3.  How  they  store  information? 

DEFINITION  -  No  matter  to  say  that  everyone  of  us  has  a  well  defined  idea  of  what  a 
CNN  is.  But  it  is  to  notice  that  the  many  contributions  that  appeared  in  the  literature  after 
the  first  papers  describing  the  pioneering  woric  carried  out  at  the  University  of  California, 
Berkeley,  generally  do  not  comply  with  the  definition  given  in  those  early  papers.  So  we  will 
try  to  investigate  in  a  systematic  way  which  items  are  essential  properties  of  CNN’s.  As  a 
conclusion  it  seemed  us  logic  to  identify  the  following  three: 

1.  A  regular  array  (not  necessary  bidimcnsional)  of  identical  cells. 

2.  Links  limited  to  a  very  restricted  neighborhood. 

3.  Dynamics  of  each  cell  described  by  a  nonlinear  differential  equation. 
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BEHAVIOR  -  Regardless  of  the  particular  kind  of  neural  network  under  consideration, 
we  may  look  at  it  as  to  a  black  box  with  an  input  and  an  output  Its  purpose  is  to  process  the 
information  contained  in  the  input  signal  in  order  to  obtain  some  specific  result  as,  e.g.,  to 
recognize  patterns,  to  classify  data,  to  detect  moving  objects,  to  solve  a  problem  of  minimum, 
etc.  This  means  that  the  network  maps  the  space  of  all  the  admissible  inputs  into  the  space 
of  its  outputs  according  to  some  well  defined  rule. 

In  the  said  operation  we  may  notice  that  many  different  circumstances  can  occur,  such 
as,  e.g.: 

1.  The  network  is  a  dynamic  system  described  by  a  (vector)  differential  equation  (au¬ 
tonomous  or  non  autonomous). 

2.  The  network  implements  an  equation  that  can  be  explicitly  solved  with  respect  to  the 
output,  that  turns  out  to  be  a  function  or  a  functional  of  the  input. 

3.  Parameters  of  the  equation  may  be  constant  or  not. 

4.  Input  signal  may  be  fed  through  the  initial  state,  or  the  forcing  function,  or  the  parameters. 

5.  Output  may  be  drawn  in  many  ways.  If  the  output  is  an  attractor,  it  may  be  of  different 
kinds,  even  strange. 

We  will  try  to  pick  up  connections  between  the  different  ways  of  operation  and  the 
definition  of  CNN’s,  as  far  as  it  is  relevant  to  their  behavior  in  storing  and  processing 
information.  INFORMATION  -  Programmability  is  one  of  the  most  attractive  properties 
of  CNN’s,  but  how  to  choose  the  most  suitable  network  and  how  to  program  it  in  order  to 
perform  a  given  task,  is  still  an  open  question.  A  step  towards  its  solution  could  be  to  have 
an  insight  into  the  kind  and  the  amount  of  information  that  a  CNN  is  able  to  store.  In  fact 
it  is  well  known  that,  due  to  the  distributed  way  of  storing  and  processing  information,  the 
organized  structure  of  a  neurocomputer  adds  significant  relations  to  data  fed  to  it.  E.g.,  in 
the  ca.se  of  programming  by  examples,  this  is  the  reason  why  it  is  able  to  generalize  from 
a  limited  number  of  training  inputs. 

Elsewhere  we  introduced  the  concept  of  "relational  information"  as  a  peculiar  property 
of  neurocomputers  and  suggested  methods  for  measuring  it.  As  a  conclusion  of  this  talk  we 
will  start  discussing  how  such  a  concept  can  be  applied  to  CNN 
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ABSTRACT 

In  this  paper  some  stability  properties  of  cellular  neural  networks  with  delay  (DCNN’s) 
are  studied;  reciprocal  and  non-reciprocal  DCNN’s  are  considered.  It  is  shown  that  a 
S3'mmetric  DCNN  can  become  unstable  if  the  delay  is  suitably  chosen;  moreover  a 
sufficient  condition  is  presented  to  assure  the  complete  stability:  such  a.  condition 
establishes  a  relation  between  the  delay  time  and  the  parameters  of  the  network.  A 
sufficient  condition  for  the  complete  stal)ility  is  also  given  for  non-reciprocal  networks: 
such  a  condition  is  independent  from  the  delay  time  and  depends  only  on  the  cloning- 
template  and  the  delay-cloniug-template. 


1.  INTRODUCTION 


C^ellular  neural  networks  (CNN’s),  introduced  by  L.  0.  Chua  and  L.Yang  in  1988  [1], 
[2],  have  found  important  applications  in  signal  processing,  especially  in  static  image 
treatment.  The  stability  of  such  networks  has  been  investigated  in  [1],  [3]  and  [4]: 
in  [1]  it  has  been  proved  that  symmetric  CNN’s  are  completely  stable;  in  [3]  a  weaker 
property  (stability  almost  everywhere)  has  been  established  for  the  class  of  the  positive 
cell  linking  templates  and  this  result  has  been  extended  in  [4],  by  means  of  ecjui valent 
t  ransfor  mat  ions . 

Processing  of  moving  images  requires  the  introduction  of  delay  in  the  signals  transmit¬ 
ted  among  the  cells  (DCNN’s)  [5].  The  study  of  stability  in  this  case  is  much  more 
difficult  than  for  conventional  CNN’s.  In  [6]  it  has  been  proved  that  positive  cell  link¬ 
ing  templates  are  stable  almost  everywhere.  In  this  paper  we  use  the  technique  of 
the  Lyapunov  functionals  to  study  the  dynamic  behaviour  of  two  types  of  DCNN’s. 
Firstly  we  consider  a  symmetric  CNN  and  suppose  to  introduce  a  delay  among  the 
cells;  we  will  prove  that  a  symmetric  CNN  with  delay  can  become  unstable  if  the  delay 
is  suitably  chosen  and  we  will  give  a  sufficient  condition  in  order  to  ensure  complete 
stability.  This  condition  relates  the  parameters  of  the  network  to  the  introduced  delay 
r.  Then  we  study  a  general  non  reciprocal  DCNN,  with  the  constraint  that  any  signals 
interchanged  between  a  couple  of  cells  is  delayed,  while  the  feedback  of  a  cell  on  itself  is 
not  delayed:  in  this  ca.se  we  derive  a  sufficient  condition  to  ensure  the  complete  stability 
of  the  network. 
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2.  CELLULAR  NEURAL  NETWORKS  AND  CELLULAR  NEURAL  NETWORKS 
WITH  DELAY 

In  order  to  study  the  stability  properties  of  a  DCNN,  let  us  compare  the  state  eciuatioii 
of  a  CNN  (Cellular  neural  networks  without  delay)  with  the  one  of  a  DCNN  (Cellular 
neural  networks  with  delay).  Given  &  M  x  N  CNN,  after  having  ordered  the  cells  in 
some  way,  the  state  ecjuation  can  be  written  as  follows: 

X  =  —X  +  Ay  +  Bn  ( 1 ) 


where: 

.V,  .i-  €  are  the  state  vector  and  its  derivative; 

y  €  is  the  output  vector; 

u  €  is  the  input  vector; 

A,  B  £  n-jatrices,  derived  from  the  feedback  template  and  from  the 

input  template  [.5]. 

E<iuations  ( 1 )  is  a  system  of  differential  equations  of  the  type: 

.f  =  f{x) 

where  /  is  a  mapping  from  R^^^^  to 

The  state  equations  of  a  r-delayed  DCNN,  by  proper  ordering  of  the  cells,  can  be 
written  as: 

x{t)  =  -x{t)  +  Aoy{t)  +  Aiy{t  -t)  +  Bu  (2) 

w’here  Ao,  Ai,  B  €  are  matrices,  derived  from  the  feedback  template,  the 

delay  cloning  template  and  the  input  template  [5]. 

Equation  (2)  is  a  particular  case  of  the  general  functional  differential  equation  [11]: 

i  =  fi^t) 

f  €  C’(C'{[— T,  is  a  continuous  mapping  of  the  space  of  functions  Xt 

into 

We  study  the  stability  properties  of  DCNN’s  described  by  ecjuation  (2).  To  simplify 
the  proofs,  we  assume  that  u  =  0. 

Definition  1:  A  dynamical  autonomous  system  is  said  to  be  completely  stable  if  and 
only  if,  for  any  initial  point  in  the  state  space,  the  (unique)  forwards  trajectory  reaches 
a  stable  equilibrium  point. 

According  to  our  previous  notations,  the  state  space  is  R^^^^  for  a  system  without 
delay  and  C([-t,0],R^^'‘^)  for  a  system  with  r-delay. 

.3.  SYMMETRIC  DCNN’S 

The  complete  stability  of  a  symmetric  CNN  without  delay,  described  by  equation  (1), 
has  been  proven  in  [1]  by  introducing  a  suitable  Lyapunov  function,  i.e.  a  mapping 
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of  into  H.  When  equation  (1)  is  replaced  by  equation  (2),  such  a  funcl.ion  can 

be  replaced  by  a  suitable  Lyapunov  functional,  i.e.  by  a  mapping  of  C([— r,  0], 
into  R  [11]. 

Dffiiiitioii  2:  We  define  the  following  Lyapunov  functional: 

V{.rt)  =  y\t)Py{t)  -  jjy'(t  +  6)-  y\t)]A[mMyit  +  6)-  y{t}]de  (.3) 

where  f(9)  is  any  scalar  function  continuous  with  its  derivative  on  [— t,  0],  /  € 
and  P  =  —I  +  Ao  +  Ai,  A\  is  the  transpose  of  A\. 

By  means  of  such  a  functional,  we  can  find  a  sufficient  condition  for  the  complete 
stability  of  DCNN’s  in  terms  of  the  euclidean  norm  of  .4i,  which  we  denote  as  jjAiK, 
and  of  the  delay  r. 

Theorem  1:  If  Ai  is  invertible,  P  is  symmetric  and: 

Mill  <  ^  (■!) 

then  the  corresponding  DCNN  is  completely  stable. 

Proof:  see  [12]. 

Moreover  it  is  possible  to  find  a  symmetric  DCNN  and  a  delay  r,  where  the  above 
condition  is  not  verified,  which  can  oscillate. 

Consider  the  following  dynamic  equations  of  a  (2  x  1)-DCNN: 

^  =  -xi(0  +  a^iyi(t)  +  ai2y2{t)  +  -  r)  +  -  t) 

^  =  -.T2(0  +  auViit)  +  «?iy2(0  +  'inViit  -  t)  +  a\^y2{t  -  r) 

Suppose  the  cells  are  working  in  linear  region,  i.e.; 

xit)  =  Hx(t)  +  Aix(t  -  r)  (.5) 

where 


and  /?  =  Oji  —  1  >  0. 

Thus  the  eigen\  alue  equation  for  an  arbitrary  delay  r  is: 


det{H  —  XI  +  Ai  exp(— At))  =  0 


(6) 


It  is  possible  to  show  (see  [12])  that  by  choosing: 


/i  >  0 

0  <  tt?.,  <  - 

T 

rtjj  <  0 


the  eigenvalue  equation  has  one  imaginary  solution,  while  all  the  other  infinite  solutions 
have  negative  real  part:  this  means  that  the  system  admits  of  a  stable  closed  orl)it  and 
thus  can  oscillate.  In  [12]  it  is  shown  that  in  this  case  the  condition  stated  in  Theorem 
1  is  not  verified. 

If  all  the  parameters  of  the  template  and  of  the  delay-template  are  positive,  the  sys¬ 
tem  admits  of  a  closed  orbit,  but  this  latter  is  not  stable  (this  is  the  case  of  the 
cooperative-irreducible  DCNN,  dealt  with  in  [6],  where  the  stability  almost  everywhere 
can  be  assured). 


4.  NON  RECIPROCAL  DCNN’S 

Let  us  consider  a  DCNN,  described  by  state  equations  (2),  where:  H  =  Ao  —  l  =  hi  is 
diagonal,  Ai  is  a  general  matrix  (even  non  symmetric)  and  ti  =  0:  this  means  that  all 
the  signals  interchanged  between  the  cells  are  r-delayed,  while  the  feedback  of  a  cell 
on  itself  is  not  delayed. 

Theorem  2:  If  the  diagonal  element  of  /f ,  h,  is  greater  than  the  sum  of  the  moduli  of  the 
elements  of  the  delay-cloning-template,  then  the  corresponding  DCNN  is  completely 
stable. 

Proof.  Let  us  introduce  the  following  Lyapunov  functional: 

V{xt)  =  x\t)x{t)  —  j  x'(t  +  9)  c  x{t  +  9)d6  (7) 


where  c  is  a  scalar  constant. 


Suppose  the  system  starts  from  an  initial  condition,  .to(^)  6  C([— r. 0],/?^^^^)  9  € 
[— r,  0],  •’uch  that  for  any  9  €  [— 7"i0]  all  the  components  of  .tq  are  less  than  1;  i.e. 
all  the  cells  are  working  in  the  linear  region.  Note  that  if  a  cell  reaches  a  saturation 
region,  under  the  hyphotesis  of  theorem  2,  it  cannot  leave  this  latter;  thus  the  above 
hyphotesis  can  be  done  for  the  remaining  cells. 

The  derivative  of  V'(.jq)  yields  the  following  expression  until  all  the  cells  are  working 
in  the  linear  region: 


Vixt)  =  [a:(0,.T(<  -  t)] 


'  2H-C  ■ 

x(t) 

A\  c 

x{t  —  t) 

(8) 


The  above  expression  satisfies  the  following  inecjuality: 


(9) 


■2/:-c 

\-rit)\ 

l.r(<  -  r)| 

The  positivity  of  V’  is  assured,  by  Silvester  test,  if: 


‘2h-c>0  (10) 

c^-2/ic+11/1i|12  <0  -»/i>pi||.  (11) 

It  is  easy  to  show  that  an  upper  bound  of  the  norm  of  Ai  is  given  by  the  sum  of  the 
absolute  values  of  the  elements  of  the  delay-clouing-template,  which  iii  the  following 
will  be  indicated  as  S.  By  choosing  a  suitable  value  of  c,  the  condition  h  >  S  assures 
that  the  derivative  of  the  Lyapunov  functional  is  positive  everywhere  and  this  means 
that  at  least  one  cell  will  reach  the  saturation  region.  Since  h  >  S,  if  the  state  of  a 
cell  Xi  reaches  the  value  1  or  —1,  it  remains  in  the  saturation  region,  regardless  of  the 
v'alues  of  the  state  of  the  other  cells;  thei’efore  such  a  cell  becomes  a  constant  input  for 
the  system.  In  fact  by  supposing  that  the  cell  i  reaches  a  saturation  region,  the  state 
equation  of  the  system  can  be  written  as: 


.i*‘  =  hlx\t)  +  -  t)  +  ti‘ 


(12) 


where: 


x\t)  = 


/  hit)  \ 
hit) 


d'i-iit) 

x<+i(0 


\  XMxNit)  ) 


(13) 


A'l  is  obtained  from  Ai  by  deleting  the  i-th  row  and  the  i-th  column;  the  constant  term 
Ui  is  obtained  by  multiplying  the  i-th  column  of  Ai  by  the  saturation  value  of  the  i-th 
cell,  y,  =  ±1.  By  putting 


z\t)  =  x\t)  +  (hi  + 

the  state  ecpiation  of  the  system  assumes  the  form: 


(14) 


i‘(0  =  hlz\t)  -f  A[h'(t  -  t)  (1.5) 

By  using  the  same  Lyapunov  functional  shown  above  it  is  possible  to  show  that  another 
cell  must  reach  the  saturation  value  and  this  procedure  can  Ije  api>lied  until  all  the  cells 
have  reached  the  saturation  region.  From  this  latter  consideration,  by  using  the  same 
arguments  of  [1],  the  complete  stability  of  the  system  can  be  derived.  Note  that  the 
etsbilished  condition  of  stability  h  >  S  is  valid  also  when  a  constant  input  is  introduced 
in  the  state  equation  (2).  QED 
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Abstract 

Since  a  Cellular  Neural  Network  (CNN)  is  a  nonlinear  analog  circuit, 
its  solutions  may  have  a  complex  dynamical  behavior,  from  complete 
stability  to  chaos.  This  paper  will  show  that  this  behavior  may  depend 
strongly  on  the  boundary  conditions  set  at  the  borders  of  the  finite- sized 
CNN,  the  network  being  stable  with  some  boundary  conditions  and  un¬ 
stable  with  others.  We  will  then  distinguish  two  kinds  of  completely 
unstable  CNNs:  those  that  are  unstable  because  of  the  boundary  condi¬ 
tions  and  those  that  are  unstable  because  of  the  template  that  defines 
them  regardless  of  the  boundary  conditions. 


1  CNNs  with  boundary  conditions 

The  architecture  of  CNNs  is  described  in  [1].  We  will  consider  CNNs  defined 
by  a  space-invariant  A-template,  without  a  B-template  and  with  1  =  0.  They 
are  therefore  described  by  the  state  equation 

r 

~  "h  Am.n  J/i+mJ+n(f) 

m,n=— r 

and  the  output  equation  yij{t)  =  f{xij{t)),  where  the  piecewise-linear  function 
/(•)  is  given  by  f(v)  =  |  (jv-f  1|  —  |v  — 1|)  and  r  is  the  size  of  the  neighborhood. 

We  will  surround  the  rectangular  array  with  boundary  cells,  whose  outputs 
lie  between  —1  and  1  and  remain  constant  with  respect  to  time  t.  The  width  of 
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this  frame  of  boundary  cells  is  equal  to  r.  Their  outputs  will  be  referred  to  as 
the  boundary  conditions  (Up  to  now,  in  most  applications  using  CNNs,  these 
boundary  conditions  have  been  set  to  zero  most  of  the  time). 


2  Two  kinds  of  completely  unstable  CNNs 

A  CNN  is  completely  stable  if  every  initial  state  converges  to  an  equilibrium 
[2].  Likewise,  a  CNN  will  be  said  to  be  completely  unstable  if  no  initial  state 
(except  possibly  a  set  of  measure  zero)  does  converge  to  an  equilibrium. 

Consider  first  a  CNN  with  a  neighborhood  of  size  1  (and  cells  arranged  on 
a  squared  or  hexagonal  grid).  Then  we  have  the  following  result; 

Theorem  1  A  CNN  whose  neighborhood  size  is  1  always  has  a  stable  equilib¬ 
rium  provided  the  boundary  conditions  are  correctly  chosen. 

For  such  a  CNN,  one  can  indeed  show  that  at  least  one  of  the  four  configurations 
shown  in  figure  1  is  always  a  stable  equilibrium  of  the  network.  A  black  pixel 
corresponds  to  an  output  of  1  while  a  white  pixel  corresponds  to  an  ouput 
equal  to  -1 .  This  theorem  is  a  corollary  of  a  theorem  proven  in  [3],  which  states 
that  a  CNN,  with  an  array  of  cells  of  infinite  size  and  an  A-template  such  that 
Em  and  n  even  ^m,n  >  1,  has  always  a  stable  equilibrium. 

Consequently,  if  a  CNN  defined  by  a  3  x  3  template  is  completely  unstable, 
we  can  modify  these  boundary  conditions  (by  picking  one  of  the  four  sets  of 
boundary  conditions  represented  in  figure  1),  so  that  this  CNN  does  not  remain 
unstable  any  longer.  This  implies  that  the  complete  instability  of  such  a  CNN 
is  always  due  to  the  boundary  conditions.  It  is  possible  however  to  find  CNNs 
with  a  neighborhood  of  size  2  that  are  completely  unstable,  independently  of 
the  boundary  conditions: 

Theorem  2  A  one- dimensional  CNN,  of  at  least  37  cells,  and  defined  by  a 
template 

A  =  [  A-2  A-i  Aq  Ai  A2  ] 

where  Ao  >  1  and 

A-2  +  A-i  +  Aq  +  y4i  +  A2  <  1 

A-2  —  A-\  +  Aq  —  y4i  +  A2  <  1 

A-2  —  A_1  -{■  Aq Ai  —  A2  <  1 

— i4_2  —  A-i  -f  j4o  +  Ai  —  /I2  <  1 

—A-2  —  A-i  Aq  +  Ai -{■  A2  <  1 

is  completely  unstable,  regardless  of  the  boundary  conditions. 


Figure  1:  One  of  these  four  configurations  is  always  an  equilibrium  of  a  CNN 
with  a  neighborhood  size  equal  to  1. 


The  instability  of  this  CNN  is  thus  no  longer  due  to  the  boundaxy  conditions 
but  is  a  characteristic  of  the  template  itself. 

This  theorem  is  also  a  corollary  of  a  more  general  theorem  proven  in  [3]. 


3  Examples 

Example  1.  The  one-dimensional  CNN  defined  by  the  template 

A  =  [  -s  p  s  ] 

is  known  to  be  completely  unstable  ifO<p— l<s  and  if  the  two  boundary 
conditions  at  both  ends  of  the  linear  array  are  set  to  zero  [4].  Nevertheless, 
since  A  is  a  3  x  1  template,  there  exists  a  stable  equilibrium  if  the  boundary 
conditions  are  properly  chosen,  because  of  theorem  1.  In  fact,  this  happens 
when  the  boundary  conditions  are  set  to  1  or  -1,  as  pointed  out  in  [4]  and  [5]. 
Moreover,  this  CNN  can  be  proven  to  be  completely  stable  with  such  boundary 
conditions,  if  its  number  M  of  cells  is  less  or  equal  than  three.  For  M  =  1  and 
M  =  2,  this  can  be  easily  proven.  Take  M  =  3  and,  for  instance,  the  boundary 
conditions  yo  =  1  and  j/4  =  —1.  The  state  equations  describing  this  CNN  are 
thus 


Xi(t)  = 

-Xi(<)  +  pf{Xi(t))  -b  Sf(x2{t))  -  S 

(1) 

X2(<)  = 

-X2{t)  -  Sf{xi{t))  +  pf(X2{t))  +  Sf{x3(t)) 

(2) 

X3(<)  = 

-X3(t)  -  S/(X2(<))  +  p/(X3(0)  -  -S- 

(3) 

Since  s f{x2{t))  —  s  <  0  and  —s f{x2{t))  —  s  <  0  for  any  time  t,  corollary  2  in  [6] 
implies  that  if  a:i(<o)  <  0  at  some  time  then  there  is  some  time  to  <tQ  <  -foo 
such  that  yi{t)  —  —1  for  all  <  >  whereas  if  X3(ti)  <  0  at  some  time  <1,  then 
there  is  some  time  ti  <  t[  <  -l-oo  such  that  ysit)  =  —  1  for  all  <  >  t[.  In  these 
two  cases,  the  network  is  equivalent  to  a  CNN  made  of  two  cells,  with  boundary 
conditions  equal  to  ±1,  which  is  completely  stable. 

We  will  now  see  that  one  of  the  two  states  xi(<)  or  X3(t)  must  become 
strictly  negative  at  some  time  <2-  Suppose  on  the  contrary  that  Xi(t)  >  0  and 
X3(<)  >  0  for  all  time  t.  Then  by  adding  (1)  and  (3),  we  get 

ii(0  +  ^3(<)  =  -(3;i(<)  +  X3(t))-|-p(/(xi(t))  +  /(x3(<)))-2s  <  2(p-l-s)  <  0. 

Hence  Xi(t2)  -f  X3(t2)  becomes  strictly  negative  at  some  time  0  <  <2  <  00, 
implying  that  xi{t2)  <  0  or  X3(t2)  <  0.  Consequently  this  CNN  is  completely 
stable. 


IQl 


Example  2.  The  one-dimensional  CNN  defined  by  the  template 
y4  =  [  — s  V  p  s  0  ] 


is  also  completely  unstable  if  p  >  1,  u  >  0  and  s  >  p  —  1  v  and  if  the 
four  boundary  conditions  are  equal  to  zero,  but  has  a  stable  equilibrium  (every 
steady-state  output  being  equal  to  1)  if  the  four  boundary  conditions  are  set 
to  1. 

Example  3.  Another  way  of  introducing  "boundary  conditions”  is  to  con¬ 
nect  the  first  cell  of  a  one-dimensional  array  to  the  last  one,  making  a  ring.  If 
such  a  CNN  is  defined  by  the  template 

A  =  [  -s  p  0  ]  , 


with  s  >  p  —  1  >  0,  it  is  completely  stable  if  its  number  of  cells  is  even 
but  completely  unstable  if  its  number  of  cells  is  odd!  Note  that  without  the 
additional  connection  between  the  first  and  the  last  cell,  this  CNN  is  completely 
stable  for  any  number  of  cells,  and  for  any  values  of  the  steady  outputs  of  the 
boundary  cells,  since  this  template  is  acyclic  [7]. 

Example  4.  In  contrast  to  the  three  previous  examples  showing  CNNs 
whose  stability  depends  on  the  boundary  conditions,  the  CNN  defined  by  the 
template 


A  = 


-p  p  p  ^p  0  ]  , 


where  p  >  1  is  always  completely  unstable  if  its  number  of  cells  is  sufficiently 
large,  regardless  of  the  boundary  conditions,  since  it  satisfies  the  set  of  inequal¬ 
ities  of  theorem  2. 


4  Conclusion 

The  two  theorems  presented  in  this  paper  show  that  some  CNNs  are  completely 
unstable  because  of  the  boundciry  conditions,  and  will  not  stay  completely 
unstable  if  we  change  these  boundary  conditions  properly;  whereeis  other  CNNs 
(with  a  neighborhood  size  greater  or  equal  to  2)  are  always  unstable  regardless 
of  the  boundary  conditions. 
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Abstract 

This  paper  outlines  a  proof  for  the  convergence  of  reciprocal  Time-Discrete  Cellular 
Neural  Networks  with  continuous,  monotone  increasing  nonlinearities.  The  proof  uses  a 
Lyapunov  function  of  the  Time-Discrete  Cellular  Neural  Network. 

1  Introduction 

The  operation  of  every  Cellular  Neural  Network  (CNN)  is  based  on  dynamic  transitions  of  the 
state  variables  which  transform  a  given  initial  state  into  a  desired  output  state.  Consequently 
the  analysis  of  CNN  stability  and  convergence  is  a  very  important  topic  in  CNN  theory. 

Up  to  now  convergence  for  Time-Discrete  CNNs  with  a  threshold-type  nonlinearity  and 
eigendominant  or  single-neighbor  dominant  templates  has  been  proved  by  Harrer  emd  Nossek 
[1],[2].  Hui  and  Ziik  [3]  proved  the  convergence  of  Time- Discrete  CNNs  with  a  piecewise  linear 
output  function  and  feedback  operators  which  comply  with: 

M-lN-l  M-lN-1 

k=0  /=0  *=0  1=0 

for  all  possible  inputs  xiij. 

This  paper  outlines  a  proof  for  the  convergence  of  reciprocal  Time  Discrete  CNNs  with 
continuous,  monotone  increasing  nonlineaiities.  The  required  constraints  for  the  feedback 
operator  values  are  less  restrictive  than  those  given  in  equation  1. 

2  Architecture  of  the  Time-Discrete  CNN 

A  two-dimensional  and  reciprocal  Time  Discrete  CNN  of  M  x  N  cells  can  be  described  by 
the  following  equations: 

M-lN-l  M-lN-1 

Z)  +  Z)  Z)  +  di,j  (2) 

At=0  1=0  k=0  1=0 

for  all  t  =  0, l,--*,Af  -  1;  j  =  0, 1,  •  •  • ,  TV  -  1 


tTr 


State  Equation; 
«i,j(n  +  l)  = 
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Output  equation 


1  for  Xij{n)  >  1 

for  <  1 

-1  for  Xij{n)  <  -1 


(3) 


with 


df{x) 

0  <  <  00 

ax 


for  |a:|  <  1  and 


dfi^) 

dx 


=  0  for  |a:|  =  1 


Parameter  assumptions 
•  Reciprocity  of  feedback  operators 

=  A{kJ\i,j)  for  all  *  =  0, !,•••,  Af  -  1;  j  =  0, !,•••, iV  -  1; 

jb  =  0,l,-..,Af-l;  /  =  0,1,.-.,JV-1  (4) 


•  Constraints  for  feedback  operator  values 

A{i,j\i,j)>  X)  t  =  0,1, j  =  0, !,•••, JV-1  (5) 

fc=o  1=0 
fcjti 

Remark: 

1.  This  constraint  for  the  feedback  operator  values  is  sufficient  but  not  necessary  to 
guarantee  convergence  to  the  equilibrium  states.  It  will  be  derived  from  a  less 
restrictive  constraint  in  paragraph  5. 


3  Matrix- Vector  Formulation 

It  is  convenient  to  describe  the  dynamics  of  the  Time-Discrete  CNN  with  a  matrix-vector 
notation  which  can  be  derived  from  equations  (2)  and  (3)  by  ordering  the  output  variables  in 
a  rowwise  sequence: 


MN  MN 


X,,(n-f-l)  = 

(6) 

lt=0  ti=0 

and 

(7) 

with 

XiN+j 

= 

(8) 

YiN+i 

~  y*,} 

(9) 

A{iN  +  j,kN  +  l) 

=  A{i,r,k,i) 

(10) 

B{iN  +  j,kN  +  l) 

=  B{i,j;k,l) 

(11) 

DiN+j 

— 

for  all  t  =  0, 1,  •  •  • ,  M  -  1;  j  =  0, 1,  •  •  • ,  N  -  1; 

(12) 

*  =  0,1,---,M-1;  1  =  0,1,  •••,N-1 

(13) 

Renoark: 

1.  Observe  that  A{v,  p)  is  symmetric  (i.e.  A(uj  p)  =  A{fi,  v) )  because  of  equations  (4)  and 

(10) 
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4  Convergence  of  the  Time-Discrete  CNN 


The  proof  uses  the  following  Lyapunov  function  (bounded,  monotone  decreasing  pseudo  en¬ 
ergy  function)  of  the  Time-Discrete  CNN: 

Af.JV  /.MN  MN  \  MN 

=  -  E  9  E  +  E  +  dA  +  T.  9{y)dy 

,,—n  V  ^  ,.—n  „—i\  /  ,j—n  ''0 


-1  for  y  =  -1 

fl(y)  =  -  /"‘(y)  =  *  |y|  <  i  (i5) 

1  for  y  =  1 

\  ** 

The  boundedness  of  V(y'(n))  is  shown  in  the  same  way  as  in  [4].  The  proof  of  the 
monotonicity  deals  with  the  first  difference  of  V’(y’(n)): 


M  N  / ,  M  N  MN 

AV{Yin))  =  AY^n)  -  >!(»/,  +  E 


M‘NM‘N 


1  m  ‘ly  sn  ‘jv 

-5  E  E Anw 


i/=0  it=Q 

M  NM  N 


4  jn  JWM  /‘y-fn+l) 

"2  E  ^ L  .  3{y)dy 

^  v=Q  u=0  t/=0 


According  to  the  symmetry  of  A{u,  n)  and  the  mean- value  theorem  of  the  integration  this 
results  in: 

M  N  /M  N  M  N  \ 

AV(y(n))  =  -  X:  An(n)  E)  +  E 

i/=0  \<i=0  /i=0  / 

.  M  NM  N  M  N  ,  . 

E  E^K/^)Al^{n)  Ay^(n)-f  E  (l^(n  +  1)  - 

i/=0  fi=0  i/=0  '  ' 


with  =  y./(n)  +  7v  A  y,/(n)  and  7^  c  [0, 1]. 


Therefore: 


MN  /MN  M-N 

Ay(y(n))  =  -  E  Ay.(n)  E  +  E  +  D,-  9Uu) 

v=0  \  /j=0  /i=0 


M  NM  N 


■|  m  ’iT  jn  ’iT 

-5  E  E-^(‘'.»‘)An(n)  Ar„W 


l/=0  /i=:0 


The  proof  of  the  monotonicity  of  V’(y(n))  is  based  on  the  following  theorem  which  will 
be  derived  in  the  appendix: 
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Theorem  1  If  A{v,  pi)  is  a  symmetric,  positive  semidefinite  matrix  then  the  following  in¬ 
equality  is  valid  for  =  F',/(n)  -j-y,,  A  Yi,  with  G  [0, 1]  and  for  all  n>  0: 

MN  (MN  MN 

-  An(n)  Y.  +  E 

u=Q  \  (A=0  ii=0 

.  MNMN 

^  |E  E-^(‘'>A^)^WAy»  (17) 

^  i/=0  M=0 

where  the  identity  is  valid  if  and  only  if  AY„{n)  —  0  for  all  v. 

Inserting  equation  (17)  into  equation  (16)  results  in: 

AV(y(n))<0  (18) 

which  means  that  V{Y{n))  is  monotone  decreasing.  The  identity  is  valid  if  and  only  if 
Ay'„(n)  =  Y„(n  +  1)  -  Y„(n)  =  0  for  all  v. 

Every  bounded,  monotone  decreasing  sequence  is  convergent,  i.e. 

lim  1^(5^ (n))  =  const.  (19) 

tl^OO 

and  therefore 

Urn  Ani^(n))  =  0  (20) 

fl— *00 

According  to  equation  (18)  AV(y(n))  =  0  corresponds  to  i;,(n+ 1)  =  ^/(n)  for  all  v  and 
aU  n  which  means  that  the  Time-Discrete  CNN  converges  towards  an  equilibrium. 

5  Positive  semidefiniteness  of  A{v^  fi) 

The  proof  in  paragraph  4  requires  that  A{v,  ft)  is  positive  semidefinite.  A  sufficient  but  not 
necessary  condition  for  this  can  be  derived  from  the  theorem  of  Gerschgorin: 

Mff 

foraU  i/  =  0, !,•••, M-AT  (21) 

/i=0 

Because  of  equation  (10)  this  results  in  the  following  restriction  for  the  original  feedback 
operator  values: 

M-lN-l 

>l(i,j;i,i)>  E  EI^(*’J5*,0I  foTvIl  i  =  0,l,---,M-l;  i  =  0,l,---,N-l  (22) 

k=0  1=0 
ijij 

This  restriction  is  fuUlilled  by  most  of  the  known  feedback  operators  (e.g.  noise  removal, 
edge  and  comer  detection,  motion  detection). 
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A  Appendix:  Proof  of  Theorem  1 

The  proof  is  based  on  the  following  theorem: 

Theorem  2  AYl,!”)  =  Ti/(w  +  1)  —  has  the  same  sign  as 

MN  MN 

Y,  Y  +  i?.  -  9{i.) 

H=0  ftzzO 

i.e.  the  following  equation  is  valid  for  a// 1/  =  0, 1,  •  •  • ,  M  •  JV; 

CM  N  M-N 

Y  A{u,,i)YM  +  E  +  D,-  9Uu) 

n=0  fi=0 

with  a»/  >  0 


Proof  of  Theorem  2  : 

Inserting  the  state  equation  (2)  and  the  output  equation  (3)  into  the  first  difference  AYt,{n) 
results  in: 


(MN  MN  \ 

Y  A{u,,i)Y^in)  +  Y  +  dA-  y,(n) 

*1=0  ,1=0  / 

(MN  MN  \ 

Y,  +  E  »(■'./•)«',.  +  D.  +  j(&)  -  j(£.)  -  Y,{n) 

M=0  ,1=0  / 


The  Taylor’s  expansion  of  /(•)  at  g{^v)  is  given  by: 


(MN  MN  \  X 

£  »*)r.(n)  +  E  e(<',  -  J({v)  ^ 


Consequently  with  /  (ff(^i/))  =  =  Y„{n)  +  7^  A  Y^{n)  : 


(l-7.)Ay,(»)  =  YM-Yu{n)+(YM^^f*)YM  +  l5 

\m=0  ,1=0  /  ****' 

Therefore  with  =  1/(1  -  7^)  •  dffdz  >  0: 

(AfiV  MN  \ 

Y  ■My^l^)Yt,{n)  +  Y  +  ]  a„  q.e.d. 

M=0  M=0  / 
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Proof  of  Theorem  1: 

Case  1:  Yi,{n  +  1)  =  Yu{n)  i.e.  Ay,,(n)  =  0  for  all  i/: 

Both  sides  of  the  inequality  (17)  are  zero  and  therefore  identical. 

Case  2;  Y^(n  ^  1)  ^  Y^{n)  i.e.  Ay’i,(n)  ^  0  for  at  least  one  u: 

Therefore  according  to  theorem  2: 

M  N  MN 

0  and  ^  #  0 

n=0 


Inserting  this  into  the  inequality  (17)  results  in: 


MN 


(MN 


MN 


-  x;  Ay.(n)  Y,  +  E  +  Du-  ff(6) 


v=0 


,1^=0 


M=o 


M  N 


=  -  E  (  l5 + 15  +  Du-  5(6)  I 

i/=0  \tt=0  n=0  ) 

,  M  NM  N 

<  0  <  -  X)  E  ^  Yuin)  A  y^(n)  q.e.d. 

^  i/=0  #1=0 

the  last  inequality  is  valid  because  A(v,  fi)  is  positive  semideiinite 
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Abstract.  The  paper  presents  a  gir^h-theoretic  method  for  computation  of  the  number  of  stable  equilibrium 
points  and  analysis  of  the  structure  of  CNN  equilibrium  set.  The  considerations  are  based  on  one  dimensional 
CNN  layout.  The  total  number  of  these  equilibria,  referred  to  as  'output  capacity'  is  shown  to  be  equal  to  the 
number  of  paths  in  a  graph  derived  from  neighborhood  consistency  conditions.  It  may  vary  vastly  depoiding 
on  interaction  weights  and  bias  current  of  the  cells.  It  is  noted  that  output  capacity  equal  to  zero  implies 
oscillations  of  the  CNN  state. 


Introduction 

The  number  of  stable  equilibrium  points  is  one  of  major  determinants  of  the  signal 
processing  capabilities  of  CNN.  It  gives  the  upper  limit  of  the  number  of  possible  different 
outputs  produced  by  the  network,  i.e.  maximum  number  of  patterns  that  can  be  retrieved,  and 
is  henceforth  termed  also  "CNN  output  capacity".  The  structure  of  the  set  of  equilibria  is 
also  practically  very  meaningful  and  puts  constraints  on  the  classes  of  patterns  that  can  and 
can  not  be  expected  to  emerge  as  the  result  of  network  computations.  In  the  following,  the 
algorithm  is  presented  for  computation  of  the  number  of  different  output  vectors  and  used  to 
study  their  dependency  on  some  parameters  of  the  network. 

Notation.  The  cellular  neural  network  considered  is  composed  of  N  linearly  ordered  neurons, 
each  governed  by  the  state  equation 

dx 

=  'ZAjfixp^J^  ,  |x,(0)|il  ,  (1) 

«*  f  Jem 

in  standard  notation  [1,2];  f  is  identity  function  saturating  at  ±1;  J,  is  bias  current;  r,C 
positive  constants;  vector  A={A^,..,A0,..Ai]  denote  interaction  mask  in  neighborhood  of  size 
k.  For  brevity  we  write  x,y  instead  of  standard  Vy.  Under  relatively  mild  condition 
(Ao>l/r)  [1,3],  state  trajectories  converge  to  stable  equilibria,  given  by 

Jem 

where  l/=r/  and  tentatively  assumed  Ji=J=const.  Any  equilibrium  satisfying  |x,|  >7  for 
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every  cell  i=l..N  is  stable.  In  signal  processing  setting  CNN  transforms  the  initial  state 
x(t=0)  (input)  to  one  of  its  stable  equilibria  x(t=oo)  and  produce  stable  binary  output  vector 
y=f(x).  The  actual  transformation  implemented  depends  upon  the  selection  of  masks  A  and 
bias  currents  J  (which  may  include  the  contribution  of  masks  B  [1]).  It  was  assumed  that 
r,C=l  and  neighbourhood  radius  k=l  in  presented  examples. 

The  algorithm  of  "output  capacity"  calculations 

In  order  to  find  all  stable  equilibrium  points  of  the  CNN  for  given  mask  and  bias 
current,  we  first  determine  all  possible,  stable  neighborhLVxls  for  every  cell,  and  then 
concatenate  them  to  obtain  full  state  vector. 

In  the  first  step,  consider  the  state  xcfz  typical  cell  and  its  neighborhood.  Let  z,  = 
sign(x,)  and  z=[z.*,..Zo,..z*]  denote  a  binary  pattern  of  output  states  in  the  neighborhood  of 
a  neuron  spanned  by  mask  A,  positive  states  corresponding  to  z,  =  and  negative  to  z,=-/. 
There  exist  2^*^  different  patterns  z,  and  each  of  them  can  be  conveniently  represented  as  a 
unique  binary  number  (-1  stands  for  binary  digit  0).  We  mark  by  z*,  z'  respectively  center- 
on  (Zo=+l)  and  center-off  (Zo=-J)  neighborhood  patterns.  For  example,  if  neighborhood 
radius  k=l,  eight  different  cases  of  z  can  be  enumerated  : 
four  center-off 

0=[-,-,-]  !=[-,-,+]  4=[-H,-,-]  5=[+r,+] 

and  four  center-on 

2=[-,  +  ,-]  3=[-,  +  ,+]  6=[+, 7=[-J-,-J-,-l-]. 

The  equilibrium  conditions  (2)  allow  only  of  some  of  them  to  exist  in  a  stable  output.  To 
determine  those  configurations,  termed  "admissible  neighborhood  patterns",  we  write  local 
equilibrium  conditions  for  a  cell  and  surrounding  neighborhood  as  an  alternative 

£(z0  >  l-U  or  Eizl  <  -l-U  (3) 


where  the  quantity 


Eiz)-rY,ZjAj  (4) 

Jeff 

represents  "excitation  level"  of  the  center  neuron,  received  from  its  neighborhood,  given 
neighborhood  pattern  z. 

In  the  second  step  of  the  algorithm  we  consider  the  total  number  of  N  dimensional 
CNN  output  vectors  that  can  be  composed  of  (2k+l)  dimensional  patterns  z  satisfying 
conditions  (3).  Conceptually,  any  such  vector  can  be  built  by  putting  a  number  of  admissible 


neighborhcxxls  together  in  the  string,  starting  from  leftmost  boundary  cell  to  the  right.  If  the 
network  is  spatially  homogenous  i.e.  parameters  A,  U  are  identical  for  all  the  cells,  the  sets 
of  admissible  neighborhoods  are  also  identical;  otherwise  admissible  neighboroods  have  to 
be  recomputed  for  every  cell.  For  1 -neighborhoods  the  process  is  illustrated  below 


The  pattern  5  =  [-l— +]  can  be  followed  by  3=[-+-+]  (as  showed)  or  alternatively  by  2=[-+- 
],  but  not  by  any  other. 

In  general,  the  sequential  process  have  to  fulfill  the  constraining  requirement  that  the 

patterns  around  two  neighboring  center  cells  should  be  identical  except  the  left  boundary  cell 

of  the  preceding  and  the  right  of  the  following  one. 

This  constraint  can  be  expressed  in  the 

form  of  the  neighborhood  matching  graph 

(G).  The  nodes  of  G  correspond  to 

admissible  neighborhood  patterns  z  and  are 

marked  with  appropriate  binary  numbers. 

Two  nodes  ij  are  connected  by  a  directed 

branch  from  i  to  j  if  pattern  i  can  be 

followed  (on  the  right  side)  by  j.  The  full 

neighborhood  matching  graph  and  its  nodal 

incidence  matrix  M  for  1 -neighborhood  is 

presented  on  Fig.  1.  Note  that  it  is 

independent  on  network  parameters,  except 

of  neighbourhood  radius,  and  its  branching 

structure  results  from  the  properties  of 

binary  numbers.  If  some  nodes  are  non 

admissible  due  to  stability  requirements  (3)  Rg.  1  Full  neighborhood  matching  graph  for 
,  .  ..  ,  ,  neighborhood  of  radius  it=7  and  its  incidence  matrix. 

their  corresponding  rows  and  columns  are 
set  to  zeros. 

The  central  point  of  proposed  algorithm  is  that  the  number  of  stable  configurations  of 
a  segment  of  one  dimensional  CNN,  composed  of  N  cells,  is  equal  to  number  of  paths  of 
length  N-1  in  the  graph  G  (with  non  admissible  nodes  deleted).  By  a  graph  theoretic 
assertion  (see  e.g.  [4],  Theorem  9.10)  this  number,  is  the  sum  of  all  elements  of  nodal 


incidence  matrix  M  of  graph  G,  taken  in  fiV-JJth  power.  Writing  K=Af  we  obtain 

ij  ij 

The  number  of  equilibrium  states  with  given  boundary  neighborhoods  ij  (possibly  identical) 
is  equal  to 

The  above  holds  if  the  sets  of  admissible  neighborhoods  are  identical  for  all  the  cells; 
if  this  is  not  the  case,  then  to  each  cell  corresponds  the  separate  set  of  admissible  nodes  and 
the  expression  for  matrix  K  is  the  product  of  appropriate  reduced  matrices  Ai, 

(6) 

where  each  Af^  preserves  only  branches  from  nodes  admissible  for  the  (/-7)th  cell  to  nodes 
admissible  for  the  Jth  cell;  all  other  entries  in  full  M  are  set  to  zeros. 


Numerical  examples  and  some  implications  for  signal  processing 

Using  the  above  algorithm  the  values  of  were  computed  for  selected  cases  of 
networks  with  neighbourhood  radius  with  /4o=2,  N=  10  and  A.i,  Aj  varying  in  the  range 
(-2,2).  The  output  capacity  of  such  a  family  of  networks  (Fig.  2)  may  assume  one  of  fixed 
values  (4,24,466,4096)  and  thus  the  numbers  of  CNN  equilibria  may  differ  by  several  orders 
of  magnitude  depending  on  the  values  of  interaction  parameters  A.  The  maximum  of  output 
capacity  is  attained  for  both  interaction  weights  close  to  zero  (middle  part  of  the  picture);  the 
second  largest  value  (466)  is  attained  for  approximately  symmetric  weights.  With  increasing 
N  one  observes  combinatorial  explosion  of  capacity  due  to  multiple  loops  in  G. 

The  actual  input  -  output 
transformation  implemented  by  CNN 
depends  upon  interaction  mask  A  and  bias 
U.  With  Ui=U  and  fixed  A,  parameter  U 
controls  the  number  of  stable  equilibria  and 
their  domains  of  attraction,  thus  changing 
the  structure  and  number  of  possible 
outcomes  of  network  computations  and 

possibility  to  distinguish  between  different  Rg.  2  Logj(X,)  (vertically)  versus  weights  A.,^, 

inputs.  As  demonstrated  by  numerous  ^(-2.2)(horizontaIpIane);^o=2;l/=0;A^=10;viewfirom 

above  (0,0). 

simulations  [2],  the  same  mask  may  give 

different  outputs  depending  on  the  bias  U.  Considering  possible  paths  in  the  graph  G  allows 
for  analysis  of  this  influence  and  prediction  of  some  aspects  of  the  network  behavior  without 


time  consuming  simulations  and  tests.  As  an  example  consider  the  mask  A =[7,2, 7].  The 
admissible  nodes  and  numbers  of  equilibria  are  presented  in  Tab.  1 . 

On  the  base  of  analogy  with  Table  I  Capacity  and  admissible  neighborhoods  for  A=[l,2,l]. 
classical  FIR  filters,  it  is 
expected  to  result  in  state  x(oo) 
being  a  low  pass  filtered  and 
thresholded  variant  of  input  i.e. 
the  states  of  neighboring  cells 
tend  to  be  of  the  same  sign, 
smoothing  out  minor 
discontinuities  and  noise  in  the 
input.  One  sees  that  this  can 
take  place  only  for  U  in  (-1,1) 
because  only  there  both  "flat" 
patterns  0=[ — ]  and  7=[-f +  -1-]  are  admissible  and  connected.  Otherwise  ,  every  input  x(0) 
gives  either  extremely  ragged  or  saturated,  flat  output.  Note  that  although  both  0  and  7  are 
admissible  also  for  U  in  (-3,-1)  or  (1,3)  there  is  no  path  between  them  so  they  can  not  coexist 
in  single  state  vector  (comp.  Fig.  1).  The  state  space  is  partitioned  onto  different 
domains,  each  leading  to  a  different  output. 

Generally,  it  may  be  expected  that  if  the  application  puts  some  constraints  on  the 
structure  of  desired  outputs,  in  the  similar  manner  the  range  of  U  can  be  inferred  from  the 
analysis  of  the  graph  of  admissible  neighborhoods  and  resulting  capacities. 

The  mapping  of  input  into  an  equilibrium  fails  if  output  capacity  equals  to  zero.  In 
fact,  K^=0  implies  oscillations,  as  the  CNN  state  trajectory  is  bounded  and  there  are  no  stable 
equilibria  (the  algorithm  counts  critical  points  of  (1)  with  all  cell  states  outside  unity,  but 
Ao>l/r  assures  instability  of  any  others).  This  situation  usually  arises  due  to  constraints 
introduced  by  network  boundary  cells,  splitting  the  admissible  nodes  into  non-connected 
subsets. 
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Abstract 

Space-varying  Cellular  Neural  Networks  (CNN's)  are  characterized  by  nearest-neighbor 
interconnections  represented  by  a  cell-dependent  connectivity  matrix  Aij.  This  biologically- 
inspired  architecture  has  many  potential  applications.  In  the  paper  an  associative  memory 
design  is  presented. 


I.  INTRODUCTION 

Cellular  Neural  Networks  represent  a  powerful  approach  to  parallel,  nonlinear  signal 
processing,  that  can  be  exploited  for  accomplishing  various  types  of  computational  tasks.  The 
simple  architecture,  with  local  connectivity,  and  the  analog  nature  of  computation  make  it 
feasible  the  VLSI  realization  of  large  networks.  The  basic  model  introduced  by  Chua  and  Yang 
[1,2]  is  characterized  by  linear  interactions  among  the  different  cells.  The  interactions  are 
represented  by  a  space-invariant  and  time-invariant  feedback  operator,  therefore  called  "cloning 
template".  Several  generalizations  of  this  basic  model  have  been  described  in  the  recent 
literature.  In  [3]  CNN's  with  nonlinear  and  delay-type  template  elements  have  been 
introduced.  In  [4]  an  adaptive  cloning  template  is  used  to  perform  a  signal  processing  task.  In 
the  present  paper,  the  space-invariant  property  is  removed.  The  resulting  CNN,  named  space- 
varying  in  the  following,  is  characterized  by  nearest-neighbor  interconnections  represented  by 
a  cell-dependent  connectivity  matrix  Ay  .  This  assumption,  which  is  a  reasonable  one  for 
biological  neural  networks,  leads  to  an  extra  complication  in  the  VLSI  realization.  However, 
the  space-varying  condition  extends  the  applicability  of  CNN's  to  a  wide  variety  of  problems. 
In  this  paper  space-varying  CNN's  are  used  :o  design  a  cellular  associative  memory.  In  an 
associative  (or  content-addressable)  memory  (AM)  the  information  is  retrieved  from  a 
corrupted  or  incomplete  version  of  it  used  as  a  key.  Associative  memories  can  be  realized  by 
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using  dynamical  neural  networks  [5].  A  stored  pattern  corresponds  to  an  equilibrium  state  of 
the  network:  the  dynamic  behaviour,  starting  from  states  sufficiently  close  (in  the  Hamming 
sense)  to  the  equilibrium  point,  will  converge  to  it,  giving  an  error  correction  capability. 
Several  design  examples  have  been  reported  that  make  use  of  full-connection  neural  networks 
(see  [6]  for  a  survey).  The  design  of  CNN's  to  function  as  associative  memories  has  been 
addressed  in  a  previous  work  [7].  However,  the  approach  followed  in  [7]  is  based  on  the 
classical  Hebb  rule.  It  is  well-known  that  a  full-connection  Hebbian  associative  memory  has  a 
very  low  capacity,  unless  the  patterns  to  be  stored  are  mutually  orthogonal.  Since  it  is 
reasonable  that  the  memory  capacity  decreases  as  the  number  of  interconnections  decreases,  a 
different  design  method  must  be  found.  In  the  following,  the  learning  algorithm  proposed  by 
Zou  et  al.  [8]  is  applied  to  compute  the  connectivity  matrices  Ay  ,  i  =1,...,  M,  j  =1,...,  N  of  an 
MxN  CNN  used  as  associative  memory.  The  details  of  the  model  and  of  the  learning  algorithm 
are  described  below. 


n.  CNN  MODEL 


With  reference  to  [1],  the  normalized  CNN  equations  are 


dv. 


^  =  -Vxij  +  Au*Vy 


i  =1,...,  M,  j  =1 . N 


(1) 


where  v^  y  represents  the  state  variable  of  cell  C(i,  j).  The  compact  notation  Ajj*Vy  represents 
a  spatial  convolution  and  is  defined  as  follows: 


Ay*Vy—  ^  Aij,JclVyJcl 

C(k.D€HCi.^ 


(2) 


Ay  is  the  feedback  operator,  C(k,l)  denotes  the  cell  on  the  kth  row  and  1th  column,  in  a 
rectangular  array.  Nj(ij)  represents  the  r-neighborhood  of  cell  Cfij);  a  value  r  =1  will  be 
assumed  throughout  the  paper,  v^jj  is  the  output  variable.  Note  that  the  control  operator  B  and 
the  threshold  I  are  not  used.  The  input-output  transfer  characteristic  of  a  cell 
is:  Vy  ij  =  0.5  0  Vx  ij  -*•  1 1  - 1  Vx  ij  - 1 1)  . 

The  symmetry  assumption  is  made: 


Aij,kl  =  ij  (3) 

for  every  i,  j,  k,  and  1. 

Condition  (3)  guarantees  the  complete  stability,  as  shown  in  [1].  It  is  well  known  that,  if  Ajj  ^ 
>  1,  the  only  observable  d.c.  solutions  correspond  to  binary  outputs,  i.e.  v  ..  -»-*-l  or  -1  ,  for 


all  i  and  j,  as  t  ^  oo.  Moreover,  it  can  be  shown  [8]  that  an  asymptotically  stable,  bipolar 
solution  Vyjj  e  {-1,  +1},  i  =1,...,  M,  j  =1,...,  N,  exists  iff  it  is 

Ay*Vy-l>0  ifvyij  =  +l  (4a) 

-Au*Vy-l>0  ifvyij  =  -l  (4)3) 

for  i  =1,...,  M,  j  =1,...,  N. 

Conditions  (4)  are  necessary  and  sufficient.  So  they  can  be  used  either  to  check  if  a  bipolar 
vector  is  a  network  solution  or  to  determine  the  network  parameters  in  order  to  have  a  desired 
solution.  In  the  following,  they  will  be  used  as  a  design  tool. 

m.  ASSOCIATIVE  MEMORY  DESIGN  USING  LINEAR  RELAXATION 

The  design  of  an  associative  memory  based  on  a  space-varying  CNN  consists  in  the 
determination  of  the  connectivity  matrices  Ay  eR^*^,  i  =1,...,  M,  j  =1,...,  N,  so  as  to  satisfy 
the  following  requirements: 

(i)  System  (1)  will  have  no  periodic  or  chaotic  solutions. 

(ii)  A  desired  set  of  matrices  Vy^*'^  e  {-1,  +1}MxN  p,  correspond  to  as  many 

asymptotically  stable  states  of  the  network. 

The  symmetry  condition  guarantees  point  (i).  Point  (ii)  is  satisfied  by  imposing  the  constraints 
(3)  for  the  desired  P  patterns,  assuming  they  are  consistent.  This  leads  to  a  set  of  linear 
constraints  that  can  be  solved  by  several  efficient  methods,  such  as  numerical  relaxation  or 
linear  programming.  The  method  we  followed  is  based  on  the  linear  relaxation  algorithm  [8]. 
Taking  into  account  the  symmetry  property,  a  link  between  two  adjacent  cells  is  characterized 
by  only  one  strenght  value.  Assuming  the  self-connection  Ay  is  chosen  a  priori,  the  number 
of  different  parameters  to  be  computed  is  easily  determined:  it  is  L  =  (N-1)  M  +  (M-1)  N-t-  2 
(N-1)  (M-1).  Hence,  PxMxN  simultaneous  linear  constraints  must  be  satisfied  in  order  to 
obtain  the  L  unknowns.  Let  S  be  the  solution  space. 

Let  us  introduce  the  following  formalism: 

Ao  is  the  common  self-feedback  value  (the  only  restriction  is  Aq  >  1); 

^=[  All, 12  Aii,2i  Aii^2 . 1^  is  vector  of  unknowns; 

Xy  €  RS,  i  =1,...,M,  j  =1,...,  N,  is  a  subvector  containing  only  the  unknowns  relative  to  the 
cell  C(ij): 

Xy  =  [Aij;i.l,j.lAij;i.l,jA:j;i.l,j+l  A|j;i,j.l  Ay;i,j+1  Aij;H.l,j.l  Aij;i+l,jAij;j+l,j4.1 


Yjj^^  e  {-1,+1  )*,  i  j  =1,...,  N,  k  =1,...,  P,  is  a  vector  containing  only  the  output 

variables  relative  to  the  cell  C(i  j)  and  to  the  kth  pattern  to  be  stored,  except  for  the  term  Vy  y.: 


The  learning  algorithm  is  as  follows: 
Step  0.  Choose  X  e  R'L  arbitrarily. 


Step  1.  Fork  =1  toP 
Fori  =1  toM 


For  j  =1  to  N 

if  (Vy  ij  =1)  and  (X/  Yij«  +Ao  -1<  0) 


then 


Xij  =  Xij-2 


XjYif+Ao-l 


Y.<» 


Y« 


else 

if  (Vy  ij  =  - 1)  and  (-  XyT  YyW  +  Ao  -1<  0) 


then 


Xij  =  Xij 


xTyf-Ao-n 


Y« 


Step  2.  if  X  €  S  then  stop 
else  goto  Step  1. 


IV.  SIMULATION  RESULTS 

In  Fig.  1  a  simulation  example  is  shown.  The  rightmost  images  represent  the  six  patterns  to  be 
stored  in  the  memory  realized  by  a  6x6  CNN.  The  self-feedback  is  Aq  =2.  All  the  unknowns 
are  initially  equal  to  2.  The  algorithm  described  above  was  utilized  to  compute  the  L=110 
different  connection  values.  The  algorithm  converged  after  105  iterations.  At  this  point  the 
network  dynamics  was  simulated  by  numerical  integration  of  Eqns.  (1).  The  leftmost  images 
in  Fig.  1  represent  corrupted  versions  of  the  stored  images  (on  the  right).  Using  these  noisy 
images  as  initial  states,  the  desired  output  image  is  obtained  as  the  final  state  of  the  network. 
Even  if  the  considered  example  is  characterized  by  modest  dimensions  (36  cells),  the 
simulation  results  are  very  encouraging.  In  fact,  the  ratio  of  the  number  of  stored  patterns  with 
respect  to  the  number  of  cells,  equal  to  0.167,  and  the  ratio  of  the  number  of  stored  patterns 


with  respect  to  the  number  of  interconnections,  equal  to  0.027,  are  quite  satisfactory  if 
compared  to  those  obtained  using  full-connected  neural  networks.  Note  that  the  present 
method  guarantees  that  the  given  set  of  patterns  are  all  stable  states  of  the  CNN,  if  S  is 
nonempty. 
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Fig.  1.  Simulation  example.  The  noisy  images  on  the  left  are  used  as  initial  states  of 
a  6x6  CNN.  The  conesponding  final  states,  on  the  right,  coincide  with  the  images 
stored  in  the  cellular  associative  memory. 
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Abstract 

In  this  paper  we  study  possibilities  of  existence  of  oscillatory  behavior  in  a  simple  closed 
loop  linear  interconnections  of  cells.  IVe  established  analytical  suffi^^l^T^t  conditions  for 
existence  of  oscillations  in  a  ring  of  any  number  of  interconnected  cells.  Simulation  expe¬ 
riments  show  very  good  agreement  with  analytical  predictions.  C\scillations  have  been  also 
confirmed  in  the  cases  of  more  complex  interconnections  of  cells  and  in  particular  in  the 
case  of  self-feedback  existing  in  all  cells. 

1.  Introduction 

Dynamic  behaviour  of  cellular  neural  networks  is  fairly  well  understood  in  cases  of 
symmetric  interconnections  [6].  However  it  has  been  pointed  out  by  many  authors  that 
even  simplest  neural  structures  can  exhibit  complex  dynamics,  typical  for  nonlinear  sy¬ 
stems,  such  as  oscillations,  fractal  basins  of  attraction  or  chaotic  behaviour  [1-4],  [7-8]. 
Even  the  simplest  two-  and  three-cell  interconnections  are  prone  to  exhibit  oscillatory  and 
chaotic  behavior  and  a  variety  of  bifurcation  phenomena  [8-9]. 

In  this  paper  we  study  the  possibilities  of  oscillatory  behavior  in  simple  cellular  neural 
networks  and  in  particular  in  closed  loop  interconnections  of  cells.  Severed  simulation  expe¬ 
riments  and  edso  laboratory  tests  have  been  carried  out  confirming  the  existence  of  oscil¬ 
latory  solutions  in  this  kind  of  structures  with  nonsymmetric  templates  or  non-uniform 
interconnections  (ie.  interconnection  templates  are  not  identical  for  all  cells).  It  is  possible 
to  show  using  rigorous  mathematical  reasoning  that  an  interconnection  of  at  least  three 
cells  is  likely  to  become  oscillatory  if  the  interconnection  weights  (or  the  cell  gains)  arc 
approprietly  adjusted.  In  particular  it  is  possible  to  show  that  asymmetric  templates  with 
no  self-feedback  give  rise  to  oscillations  when  applied  in  a  ring  structure  of  the  network. 
Influence  of  the  self-feedback  is  also  studied  in  simulation  experiments.  Presented  results 
constitute  a  contribution  to  qualitative  analysis  of  nonsymmetric  cellular  neural  netwr  ks 
and  allow  deeper  insight  into  networks  dynamics. 


0-7803-0875-1/92  3$.00  019g2IEEE 


Figure  1.  Considered  interconnection  of  cells. 

2.  Oscillation  criteria  for  closed-loop  interconnection  of  cells 


We  assunae  that  the  dynamics  of  the  i-th  cell  can  be  described  by  a  first  order  diffe¬ 
rential  equation  of  the  form; 


duiit) 

dt 


(1) 


where;  u,(t)  represents  the  voltage  across  the  *-th  storage  capacitor,  Tij  represent  the  in¬ 
terconnection  weights  (influece  of  neighboring  cells),  fj-  a  saturation- type,  monotonically 
increasing,  continuous  function. 

Let  us  assume  that  the  cells  are  interconnected  in  a  one-dimensional  array  (Fig.l). 
This  imposes  a  constraint  on  connectivity  matrix  T  =  [Tij],  The  equations  (1)  become 
{i  =  2,...N): 


dui{t) 

dt 

du.(t) 

dt 


■^[TrNVNit)  -  +  A] 


(2) 

(3) 


These  equations  can  be  rewritten  in  the  form; 


dui{i) 

dt 

dui{t) 

dt 


F’lCuyv,!'!) 

f;(u, 


(4) 

(5) 


These  equations  describe  dynamic  behavior  of  a  particular  type  of  cellular  neural  network 
-  namely  the  A  templates  are  row- vectors  with  0  self- feedback  (central  element)  and  only 
one  non-zero  element  (thus  are  nonsymmetrical). 
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The  results  presented  in  this  study  are  based  on  an  interesting  existing  result  [5]: 

THEOREM  1. 

Let  functions  F,-  satisfy  : 

F,(0,0)  >  0,  i  =  1,  ...N  and  Fi{un,0)  >  0  for  all  up/  >  0 
and  the  following  inequalities  hold  on  a  closed  subset  F  6  F”: 

IS  <  0  &  >  0  for  2  <  i  <  iV  and  Ig-  <  0. 

The  system  (5)  has  a  unique  steady  state  u*  =  [«j, in  the  set  F,  with  Fi(up/,  ui)  < 
0  if  U;v  >  u'jsj  and  ui  >  uj,  while  Fi(u^,ui)  >  0  if  ujv  <  and  u\  <  u\.  Also,  is 
bounded  above  in  F.  Let  J  =  -  the  Jacobian  matrix  of  F  at  u*.  Suppose  that  J 

has  no  repeated  eigenvalues. 

Then: 

if  J  has  any  eigenvalues  with  positive  real  parts,  the  equation  (5)  has  a  nonconstant 
periodic  solution  in  the  set  F. 

COROLLARY  1. 

If  for  AT  >  3  all  the  weights  Tij  are  zero  except  Tip/,  ...Ta-i,  i  =  2,...N  and  exactly 
one  element  among  the  nonzero  ones  is  negative  while  all  others  are  positive  furthermore 
+  7i  =  0  and  +  /,  =  0  for  i  =  2,  ...N,  then  if  the  Jacobian  matrix  satisfies 
the  condition  as  in  Theorem  1  and  the  closed  loop  of  neural  cells  possesses  a  nontrivial 
periodic  solution. 

COROLLARY  2. 

Assuming  that  all  cells  have  identical  nonlinear  saturation-type  characteristics  with  go  - 
gain  at  the  origin  the  condition  imposed  on  the  eigenvalues  of  the  Jacobian  matrix  can 
be  replaced  by: 

2 

- i - <  9o  (6) 

{-TiP/Tn...Tp/p/.i)ifcosi^) 

and  in  particular  when  =  1  as  in  the  case  of  typical  CNN  cells: 


_ 2 _ 

{-Tip/T2i...Tp/p/-i)^cos{^) 


(7) 


3.  Experimental  results 

Extensive  simulation  and  laboratory  experiments  have  been  carried  out  to  investigate 
the  dynamic  behaviors  in  ring  structures  of  cellular  neural  networks.  In  our  experiments 
we  used  a  modification  of  the  op-amp  cell  model  proposed  by  Chua  and  Yang  [10].  Below 
we  present  some  comparative  results  obtained  in  interconnections  of  3,  6  and  9  neural 
cells.  The  time  constant  of  the  cells  used  was  fixed  CR  =  10“®s.  Waveforms  preceding 
the  onset  of  oscillations  and  those  showing  how  the  oscillations  are  generated  in  the  system 
are  shown  in  Fig.2-4.  It  is  interesting  to  notice  that  the  greater  the  number  of  neurons 
in  the  ring  the  smaller  is  the  weight  value  by  which  the  oscillations  are  generated  in 
the  system.  The  frequency  of  oscillations  diminishes  with  growing  length  of  the  ring  and 
the  amplitude  of  the  state  variables  (the  capacitor  voltages)  depend  on  the  weight  value 
(template  coefficient)  and  is  bounded  as  given  by  the  formula  (3)  of  [10]  -  the  greater  the 
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t  (3) 


t  (3) 


Figure  2.  Typical  waveforms  observed  in  a  3-cell  ring  (N=3).  T,i_i  =  1.3m5'  (a)  ,  Ti,_i 
1. 47715  (b). 


Figure  3.  Typical  waveforms  observed  in  a  6-cell  ring  (N=6).  Ta-i  =  L0m5  (a)  ,  2;,_i  = 
l.lm5  (b). 


Figure  4.  Typical  waveforms  observed  in  a  9-cell  ring  (N=9).  Ta-i  =  l.OmS  (a)  ,  r„_i  = 
1.2r7i5  (b). 
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weight  value  the  greater  the  amplitude  of  the  voltages.  This  observation  might  be  crutiai 
in  the  design  as  the  upper  bound  of  this  amplitude  is  imposed  by  the  dynamic  range  and 
voltage  swing  of  the  actual  amplifier  used. 

In  further  experiments  we  tested  the  system  behavior  in  the  case  when  the  self-feedback 
coefficients  were  non-zero.  Introduction  of  a  small  self-feedback  caused  the  frequency 
of  oscillations  diminish  and  the  amplitude  grow  in  comparison  with  the  ring  without 
self- feedbacks.  The  formula  (3)  of  [10]  for  the  amplitude  bounds  still  holds. 

In  several  experiments  we  observed  that  introduction  of  self-feedback  caused  the  ne¬ 
twork  oscillate  even  in  the  case  where  without  the  self-feedback  the  system  was  allways 
going  towards  an  equilibrium  point. 

All  experiments  show  substantial  influence  of  the  particular  circuit  implementation 
used  on  the  performance  of  the  system.  Depending  on  the  actual  active  elements  used  the 
amplitude  of  oscillation  changes  and  distortion  of  the  signal  (clipping)  is  observed. 

Further  studies  are  concentrated  on  dynamic  behaviors  of  different  kinds  of  regular  in¬ 
terconnections  of  neural  cells  and  bifurcation  mechanisms  in  large  cellular  neural  networks. 

4.  Conclusions 

•  It  follows  from  Corollary  2.  that  it  is  always  possible  to  adjust  the  weights  of  cell 
interconnections  (template  coefficients)  to  obtain  oscillations  in  the  closed  loop  of 
three  or  more  cells.  It  is  possible  to  ensure  oscillatory  behavior  in  a  ring  of  odd 
number  of  identical  cells.  In  the  case  of  even  number  of  cells  at  least  one  of  the  cells 
must  have  a  different  template  (sign  of  the  weight). 

•  Similar  result  could  be  obtained  provided  the  gains  of  the  nonlinear  characteristics 
could  be  adjusted. 

•  It  is  interesting  to  note  that  the  assumption  about  the  fixed  gain  in  basic  CNN 
definition  is  very  important  as  two  systems  with  exactly  the  same  topology  and 
connection  matrices  can  have  different  dynamic  behaviours  depending  only  on  the 
gain  factors. 

•  Experiments  fully  confirm  the  theoretical  reasoning  given  above.  Morover  several 
other  types  of  oscillatory  behavior  have  been  confirmed.  These  include  parasitic 
oscillations  in  certain  realisations  of  neural  cells. 

•  Possibilities  of  existence  of  oscillatory  solutions  in  simple  CNN  structures  (as  consi¬ 
dered  above)  may  confirm  the  difficulties  in  design  of  CNN  with  templates  having 
coefficients  of  both  positive  and  negative  signs. 

•  The  circuits  discussed  in  this  paper  are  relatively  simple  however  one  should  consider 
that  similar  effects  are  even  more  likely  to  occur  in  two-  or  three-  dimensional 
interconnection  structures.  The  undesirable  effects  of  phase  shifts  as  could  be  seen 
from  our  simple  examples  are  likely  to  occure  in  the  case  of  opposite  sign  templates 
and  may  lead  to  complex  collective  behaviour  of  the  network  including  sustained 
oscillations  or  so-called  ringing  transients. 
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•  Analysis  of  CNN  from  the  point  of  view  of  nonlinear  dynamics  and  nonlinear  oscilla¬ 
tions  in  particular  should  prove  useful  in  solving  important  theoretical  and  practical 
problems  and  may  lead  to  new  areas  of  applications. 
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Abstract  -  A  generalization  of  the  Cellular  Neural  Network  paradigm  is 
obtained  by  removing  the  uniformity  constraint  on  weight  values.  Such 
Generalize  CNNs  are  capable  of  new  tasks,  such  as  function  approximation 
or  associative  memory.  A  stability  analysis  of  these  networks  13  presented. 
Adaptation  and  application  of  a  gradient  descent  leai~ning  algorithm  is  then 
discussed. 

Introduction  -  The  most  important  characteristic  of  Cellular  Neural 
Networks  (CNN)  as  defined  by  Chua  &.  Yang  [CHJ88]  is  locality  of 
connections,  that  greatly  simplifies  the  layout  problem  for  IC  realization. 

The  said  model,  however,  involves  another  significant  constraint  in 
the  uniformity  of  weight  values,  so  that  processing  consists  of  a  spatial 
convolution  with  an  operator  defined  by  the  cloning  template.  In  this  way, 
and  also  with  the  extension  to  non-linear  and  delay-type  templates  (R0S90) , 
many  image  processing  problems  have  been  successfully  solved  [CNNA90] . 
always  usirig  the  network  with  strong  enough  self-feedback  as  to  obtain 
saturated  (i.e.  ±1)  steady-state  outputs. 

If  the  constraint  on  uniformity  of  weights  is  removed,  new 
applications  may  be  conceived  for  CNNs,  such  as  Content-Addressable  Memory 
(CAM)  [TAN901 ,  classification,  function  approximation,  that  cannot  be 
implemented  by  traditional  CNNs. 

I  shall  call  Generalized  CNN  (GCNN)  a  network  which  is  identical  to 

Chua  &  Yang's  (CHU88]  when  parameters  A (i, j;k, 1 ) ,  B(i,j;k,l).  I(i.j)  are 

allowed  to  vary  arbitrarily  while  respecting  the  locality  condition.  (3CNNs 

are  included  in  the  extended  definition  of  CNN  recently  given  by  Roska 

[R(DS92] .  In  the  following,  normalization  R  =  1,  C  =  1  is  applied,  and 
kl  ^ 

B(i,j:k,l)  ■*  without  loss  of  generality.  Voltages  v^.  v^,  v^  in 
[(THUBS]  will  be  denoted  x,  y,  u  in  the  following.  I  shall  always  consider 
networks  with  clamped  inputs  and  neglect  specifying  that  sums  over  cell 
indices  must  be  taken  inside  the  significant  neighborhood. 
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Therefore,  GCNN  neuron  dynamics  is  written  as  follows: 

X.  (t)  =  -X .  . (t)  +  r  A(i.  j:k.l)f  (x. ,  (t) )  +  Ki.j)  +u 

IXie  to  the  peculiarities  of  the  model,  development  of  CNN  theory  has 
followed  methods  that  are  quite  different  from  those  usually  applied  in 
Neural  Network  (NN)  theory  and  sometimes  are  more  similar  to  Digital  Signal 
Processing  methodologies.  This  is  most  evident  concerning  one  of  the  most 
important  topics  of  NN  theory:  learning. 

In  this  paper  I  shall  discuss  stability  of  GCNNs,  based  on  known 
results  of  CNl'f  and  NN  theory.  On  this  basis  I  discuss  possible  applications 
of  the  paradigm.  The  learning  issue  is  later  confronted,  by  considering  the 
pxjssibility  of  application  of  standard  NN  algorithms,  es,'!=‘cially  a  gradient 
descent  method. 

Stability  -  It  is  pxDssible  to  apply  to  GCNNs  many  known  results  from 
CNN  and  NN  theory. 

Theorem  1  (tound  on  state  values):  If  V  i.j  lx  (0)1  <  1.  la.l  ^  1  and 

'  VJ  VJ 

II.^I  <  I.  then  all  states  are  bounded  for  all  time  t  >  0  and  the  bound 
X  is  computed  as  follows: 

max 

''max  2  +  I  +  max  |A{i,j;k.l)l 
v  .  j 

Proof:  It  is  theorem  1  of  [CHU88] . 

Theorem  2  (stability'  of  reciprocal  nets):  If  V  i,j,k,l  A(i.j;k,l)  - 
“  A(k.l;i,j),  then  the  network  is  globally  asymptotically  stable. 

Proof:  Follows  immediately  from  theorems  2,3  and  4  of  [CHU88] . 

Theorem  3  (saturated  steady-state  output):  If  V  i.j  A(i, j;i, j)  >1  then 
magnitude  of  stable  states  must  be  grater  than  1, 

Proof:  It  is  theorem  5  of  [CHU88I . 

Theorem  4  (stability  of  low-level- feedback  nets):  If  V  i.j 

A(i,j;i.j)  +  1/2  [|A(i.j;k,l)  |  +  |A(k,l;i.  j)  ij  <  1 

(where  A(i, j;i.j)  may  also  take  negative  values)  then  the  networi^  is 
globally  asymptotically  stable. 

Proof:  Follows  immediately  from  theorem  3  of  (HIR89] . 

Theorem  5  (stability  of  positive-cel  1-1  inkir^  networks):  If  V  i.j.k.l 
A(i,j;k,l)  S  0  and  V  i.j.k.l  there  is  a  path  on  the  network  grapjh  frcan  i.j 
to  k.l  pjassing  only  through  positive  weights,  then  the  networi<  is 
almost  everyvrtiere  asymptotically  stable. 


Proof:  Follows  from  theorem  1  of  [CHU90) . 


Theorem  6  (stahility  of  feed-foi-uard  ptx>cessing  GQ'Ms):  If  a  GQIN  can  be 
decomposed  into  a  cascade  of  globally  asymptotically  stable  GCNNs,  it  is 
also  globally  asymptotically  stable. 

Proof:  It  is  a  corollary  of  theorem  5  of  (HIR89) . 

Applications  -  Leaving  away  consideration  of  networks  having  periodic  or 
chaotic  attractors  (BAR92] .  two  modes  of  operation  are  interesting  for 
neural  computing:  thie  first  case  is  when,  for  every  clamped  input,  the  net 
is  globally  convergent  [HIR891  regardless  of  initial  conditions.  In  this 
case  the  net  performs  a  classification  of  inputs  when  it  has  saturated 
final  states,  or  otherwise  it  implements  a  continuous  mapping  from  inputs 
to  continuous-valued  outputs.  It  need  not  be  reset  and  can  be  cascaded  by 
transfeiTing  the  output  of  a  stage  to  the  input  of  the  following  one. 

In  the  second  mode  inputs  are  clamped  to  a  fixed  value  and  the  net  has 
multiple  attraction  basins  (it  is  convergent),  depending  on  initial 
conditions.  In  this  case  classification  or  mapping  is  done  between  initial 
conditions  and  final  outputs.  This  way  of  functioning  is  typical  of 
Content-Addressable  Memories  (CAM) . 

In  this  pap>er  I  restrict  consideration  to  networks  operating  in  the 
first  mode. 

A  task  for  such  a  network  may  be  described  in  general  by  a  function 
3^.  S’  ■*  O  where  input  space  S’  and  output  space  <0  may  be  continuous  or 
discrete.  For  instance,  in  a  classification  problem  ^  s  0?"^  and  <0  <z  Z'"  for 
suitable  n  and  m.  For  this  reason.  I  shall  discuss  learning  problems  as 
applied  to  function  approximation  tasks  for  boolean  and  continuous  mappings. 
Learning  -  All  the  work  that  is  currently  being  done  on  CNN  learning  has 
been  based  on  the  fact  that,  unlike  all  other  NN  models,  CNNs  have  very  few 
parameters  [HAR91]  (SZ091].  Tliis  is  .Tot  the  case  for  GCNNs.  and  for  this 
reason  it  is  necessary  to  exploit  classical  learning  algorithms  from  NN 
theory.  The  only  example  in  literature  may  be  found  in  ITAN90] ,  where  the 
Hebb  rule  was  used  for  CAM  purposes.  Other  possibilities  include  stochastic 
methods,  such  as  simulated  annealing,  or  gradient  descent  algorithms.  In 
this  paper  I  discuss  application  and  modification  of  a  gradient  descent 
algorithm:  Recurrent  Back-Propagation  (RBP)  (PIN87) .  which  is  a 
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generalization  of  the  well  known  Back-Propagation  to  ncrv-feed-forward 
networks . 

Recurrent  Back-Propagation  -  Consider  GCNN dynamics  (1) .  A  sigmoidal  output 
function  f  (x)  =  2(l+exp(-/?x)  )-l  was  chosen,  because  the  usual  piecewise 
linear  output  function,  having  zero  derivative  outside  the  interval  (-1.1). 
would  give  more  problems  of  local  minima  during  learning. 

Denote  the  desired  steady  state  output  matrix  with  input  matrix  u^. 
In  the  general  case,  u  and  r  take  significant  values  over  seme  units  only 
(input  and  output  neurons).  Choose  error  measure  E  as  follows: 


E  -  1/2  e;:^  - 

where  is  the  stable  state  with  input  u^,  is  1  for  output  neurons 
and  0  for  neurons  vrf-iose  state  is  hidden  to  the  external  environment. 

Gradient  descent  over  error  surface  E  yields  the  usual  delta  rule, 
v^ich  may  be  written  as: 


Aj^^^(p.q;r,s)  =■  A^(p.q;r,s)  +  AAj^(p,q;r,s) ; 

iA(p.q;r,s)  -  -»  ffllp^q.'r.s)  '  ‘ 

where  t^ .  =  f'  (x:^.)  and  Z^  is  the  fixed  point  of  the  error 

j  L  '•J  J  ''  J 

back-propagation  GCNN  with  dynamical  equation 


This  net  has  the  same  topology  as  the  original  one,  with  transposed 
wei^t  tensor,  a  linear  output  function,  and  errors  as  biases. 

System  (2)  has  the  same  fixed  points  as  (1),  with  the  same  eigenvalues 
[HER91] .  However,  convergence  of  the  back-propagation  network  is  not 
guaranteed  by  stability  of  fixed  points  of  (1) ,  because  the  output  function 
is  linear.  For  this  reason.  I  added  a  piecewise  linear  output  function 
g(x)  -  0.5(  |x+K|-|x-K| ) ;  in  this  way  we  obtain  the  same  back-propagation 
system  while  in  the  linear  region,  i.e.  in  a  neighborhood  of  the  solution 
(that  can  be  made  large  for  large  enough  K) ,  but  the  network,  vrfiose 

dynamical  equations  are  therefore  written  as 

- -Z^.  +  r,A(k,l;i,j)g(zt',  )+  E^  (2') 

V  j  X  j  t  k  I  X  j 

is  now  stable  v^enever  the  original  one  is. 
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When  learning  is  accomplished  on  a  sequential  computer,  RBP  is  rather 
slow,  because  it  needs  thousands  of  steps  involving  each  the  relaxation  of 
two  networks.  However,  this  algorithm  was  chosen  because  of  the  success  of 
ordinary  back  propagation  in  feed-forward  nets.  Thinking  of  real-life 
realizations,  RBP  may  be  implemented  in  hardware  so  that  one  iteration  step 
of  the  algorithm  lasts  only  a  few  time  constants  of  the  electronic  circuit, 
so  that  learning  time  is  actually  governed  only  by  the  frequency  of 
presentation  of  patterns. 

Simulation  results  -  Boolean  and  continuous  function  approximation  was 
tried  on  1-neighborhood  planar  networks  (type  1,  figure  1)  and  on  layered 
systems  of  one-dimensional  networks  (i.e.  planar  networks  with  selected 
connections  -  type  2,  fig.  2). 

Leai'ning  was  started  with  randcxa  weights  satisfying  theorem  4  so  as  to 
ensure  stability,  which  was  generally  preserved  during  learning,  provided 
that  the  learning  rate  was  not  too  large,  even  if  weights  eventually 
violated  the  condition. 

Type  1  nets  were  soon  discarded,  because  they  tend  to  oscillate  very 
easily  and  therefore  have  very  long  settling  times  (hundreds  of 
time-constants) . 

Type  2  networks,  instead,  proved  capable  of  approximating  boolean  and 
continuous  functions.  A  2x2  network  was  taught  to  compute  logical  AND  and 
XC®  of  its  inputs  (fig.  3);  3x3  nets  with  both  topologies  of  fig.  2(a)  and 
(b)  were  able  to  approximate  sections  of  sinusoids. 

Conclusions  and  perspectives  -  The  results  reported  in  this  work  are  enough 
to  say  that  new  applications  may  open  up  for  Cellulai'  Neui'al  Networks,  in 
the  fields  of  approximation,  reconstruction  of  signals,  classification. 
More  extensive  simulation  is  in  progress  in  order  to  inquire  into  the 
performance  of  such  networks  vrtien  confronted  with  traditional  fully 
connected  networl<s. 


figure  1 
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il  i2  f  1  (XOR)  f2(AND) 
-1  -1  -0.99  -0.97 

-1  1  0.99  -0.50 

1  -1  0.99  -0.50 

1  1  -0.96  0.94 


figure  3  XOR/AND  networic  weights  are  writteh  near  connections  and  cells: 
biases  I  are  in  brackets. 
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Abstract. 

A  possibility  of  designing  functional  devices  for  various  applications 
in  terms  of  unified  technology  on  the  basis  of  CNNs  is  discussed  on 
examples  of  an  associative  memory  device,  complex  oscillations  generator, 
devices  for  chaotic  memory  scanning  and  pattern  recognition. 

1.  INTRODUCTION. 

One  of  the  advantages  of  analog  (CNN)cellular  neural  networks  [1]  is  a 
possibility  of  their  realization  as  chips. 

But  CNNs  with  relatively  small  number  of  elements  may  also  be  promising. 
Even  if  all  the  elements  are  coupled  with  each  other,  the  total  number  of 
connections  is  not  excessive  here. 

Then  an  interesting  possibility  to  design  chips  of  integral  functional 
devices  for  various  applications  in  terms  of  unified  technology  on  the 
basis  of  CNNs  with  inhomogeneous  couplings,  appears. 

For  example,  a  possibility  to  use  neural  networks  as  A/D  converter  and 
circuits  for  solving  linear  programming  problems  were  discussed  in  [2]. 

This  approach  can  also  be  used  for  designing  various  classifiers  [3]. 

We  will  discuss  in  this  report  an  application  of  this  approach  to  the 
design  of  devices  for  chaotic  memory  scanning  and  pattern  recognition. 
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2.  ASSOCIATIVE  MEMORY,  CHAOTIC  SCANNING  AND  PACEMAKER. 


Two  devices  are  necessary  to  organize  chaotic  memory  scanning:  a  memory 
[4]  itself  and  some  external  generator  or  pacemaker  to  push  the  system  from 
one  memorized  pattern  to  another. 

2.1.  Associative  memory  on  the  basis  of  an  inhomogeneous  CNN. 

To  store  images  into  a  neural  network,  we  have  to  find  an  algorithm  of 
forming  the  matrix  of  couplings  or  templates,  providing  correspondence 
between  equilibrium  states  of  a  CNN  circuit  and  memorized  patterns. 

ADALINE  algorithm  [5]  was  chosen  as  a  learning  rule  (an  algorithm  of 
coupling  matrix  forming).  This  algorithm  was  chosen  for  the  next  reasons: 
it  is  simple  in  realization  and  its  matrix  of  couplings  is  not  symmetric. 

In  general,  the  matrix  of  couplings  is  not  local. But  demands  on  the 
locality  of  the  matrix  of  couplings  can  be  lowered  for  the  case  of  a  little 
number  of  cells  and  images.  This  case  of  a  little  number  of  cells  and  pat¬ 
terns  is  important  not  only  as  an  example,  but  for  classifier  design  also. 

2.2.  Generator  of  complex  oscillations  with  amplitude  modulation. 

The  dynamics  of  a  non-autonomous  harmonically  driven  CNN  composed  of  two 
cells  was  studied  in  details  in  [6].  A  chaotic  attractor  was  found  in  this 
non-autonomous  system. 

Such  complex  dynamics  can  obviously  be  obtained  in  an  autonomous  system, 
replacing  an  external  harmonic  signal  by  the  signal  from  a  generator 
composed  of  two  cells.  The  dynamics  of  the  four  cell-CNN  is  described  by: 

dY/dt  +  Y  =  T  *  F(Y), 

Y  =  {  y(l),  y{2),  y(3),  y(4)  ) 

F(Y)  =  {  f(y(l)),  f(y2)),  f(y(3)),  f(y(3))  ),  (1) 
f(x)  =  { jx  +  11  -  |x  -  11 )  /  2 

To  design  a  pacemaker  that  could  be  used  to  control  memory  scanning  a 
neural  generator  in  which  a  large  amplitude  chaotic  oscillations  are 
interchanged  by  oscillations  with  a  small  amplitude  near  zero  value,  should 
be  constructed. 

We  modify  then  the  system  (1)  to  a  neural  network  of  Fig.l.  Two 
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additional  controlling  cells  5  and  6  are  introduced  here.  The  dynamics  of 
the  cells  5  and  6  are  described  by  equations 

a(t)-dx/dt  +  X  =  f(u), 

a(t)  =  a^+  • sin( 2nt/T)  (2) 

where  u  is  an  output  signal  from  the  cells  1  or  3  for  the  cells  5  or  6 
respectively,  T  is  a  period  of  controlling  signal  for  parameter  o(t). 

With  an  increase  of  the  parameter  o(t)  chaotic  oscillations  disappear  in 
the  system. 

We  can  obtain  the  needed  control  signal  (Fig. 2)  for  the  memory  device 
with  the  help  of  high-pass  filter  composed  of  neural  cells. 

Applying  the  signal  from  the  pacemaker  to  the  associative  memory  device 
we  obtain  the  device  for  chaotic  scanning  of  memory.  Fig. 3  shows  the  regime 
of  memory  scanning  for  CNN  composed  of  nine  cells  with  tree  stored  patterns 
VI,  V2  and  V3. 

It  should  be  noted  that  chaotic  memory  scanning  may  also  be  organized  in 
some  other  way  [7]. 


3.  PATTERN  RECOGNITION. 

By  pattern  recognition  we  mean  the  following. Let  an  external  signal  is 
applied  to  a  neural  system,  in  which  chaotic  scanning  of  memory  is 
organized.  If  the  external  signal  corresponds  to  one  of  the  patterns, 
stored  in  the  neural  network  (memory  device), the  system  must  stabilize 
itself  in  the  state,  associated  with  that  pattern.  Otherwise,  it  should 
continue  chaotic  memory  scanning. 

For  the  organization  of  pattern  recognition  we  introduce  a  local  feed¬ 
back  between  the  memory  and  pacemaker.  This  feedback  turns  on  and  destroys 
the  complex  oscillations  of  the  pacemaker  only  if  a  given  equilibrium  state 
of  the  memory  device  corresponds  to  the  pattern  being  recognized. 

The  feedback  can  be  described  by  the  relation 

o{|Y  -  Z|)  =  k/{|Y  -  Z|  +  e  ),  (3) 

where  k  and  e  ate  some  fixed  parameters  and  e  <<  1;  vector  Z  stands  for  an 
external  signal  or  external  pattern;  vector  Y  describes  an  internal  state 
of  associative  memory;  |  *  |  is  a  vector  norm.  a(|Y  -  Z| )  is  added  to  flf(t) 
(section  2.2). 


■la? 


We  organize  pattern  recognition  only  on  one  component  of  Z. 


a(|Y  -  Z|)  =  k/( |y(2)  -  z(2)|  +  e  ),  (4) 

where  y(2)  is  the  second  component  of  Y,  z(2)  is  the  second  component  of  Z. 

Fig. 4  shows  the  trajectory  of  the  state  variable  y(2)  for  a  pattern 
recognition  regime. 


4.  CONCLUSION. 

So  we  discussed  the  possibility  of  applying  CNNs  and  their  combinations 
to  the  design  of  various  functional  devices.  This  approach  was  discussed  on 
the  examples  of  associative  memory,  complex  oscillation  generator,  chaotic 
Diemory  scanning  and  pattern  recognition.  The  advantage  of  this  approach  is 
in  the  use  of  universal  CNN  chips  for  different  purposes. 
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Fig. 3.  Chaotic  memory  scanning.  Memory  device  falls  onto  memorized 
pattern  VI  for  0  <  t  <  45,  200  <  t  <  270,  310  <  t  <  360; 

onto  V2  for  90  <  t  <  190,  280  <  t  <  300; 
onto  V3  for  45  <  t  <  90,  360  <  t  <  400. 


Fig. 4.  Pattern  recognition:  k  =  0.02,  e  =  0.002.  Z  is  an  arbitrary 
external  pattern  for  0  <  t  <  140  and  Z  =  VI  for  t  >  140. 
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Abstract 

Tlii.s  paper  covers  the  design  ol  contiiuious-time  (CT)  and  di.screte-time  (DT)  Cellular  Neural  Net¬ 
works  (CNN)  using  analog  VLSI  circuit  techniques.  A  new  cell  model  is  proposed  which  exhibits 
advantages  for  reduced  area  and  power  consumption  CNN  implementations.  This  model  is  very 
well  suited  for  implementation  in  cun'ent  domain,  which  is  also  important  to  avoid  the  need  for  cur- 
rent-to-voltage  dedicated  interfaces  in  image  processing  tasks  witli  photosensor  devices.  Cell 
design  relies  on  exploitation  of  current  mirrors  for  the  efficient  implementation  of  both  linear  and 
nonlinear  analog  operators.  These  cells  are  simpler  and  easier  to  design  than  tliose  found  in  previ¬ 
ously  reported  CT  and  DT-CNN  devices.  Basic  design  issues  are  covered,  together  with  discus¬ 
sions  on  the  influence  of  non-idealities  and  advanced  circuit  design  issues  as  well  as  design  for 
manufacturability  considerations  associated  with  statistical  analysis. 

Introduction 

Cellular  Neural  Networks  (CNN)  consist  of  arrays  of  elementary  processing  units  (cells),  each  one 
connected  only  to  a  set  of  adjacent  cells  (neighbors).  This  local  connection  property  makes  CNN 
physical  design  easy  (specially  for  the  class  of  translationally  invariant  CNNs,  where  all  inner 
cells  are  identical),  allowing  increased  cell  density  per  silicon  area.  Also,  programmability  issues 
can  be  easily  incoiporated  to  translationally  invariant  CNNs  without  significant  extra  routing  cost, 
by  just  adding  several  global  control  lines,  one  per  weight. 

CNN  properties,  and  its  application  to  image  processing,  pattern  recognition,  motion  detection, 
etc.,  have  been  covered  in  different  papers,  for  instance  [Chua88a,  88b,  90, 91a,  Noss92,  Mats90a, 
90b,  90c,  Mats91].  This  paper  focuses  on  CNN  VLSI  implementation,  of  which  little  literature  is 
available  [Cruz91,  Halo92,  Harr92].  For  implementation  purpo.ses,  analog  CNNs  can  be  classified 
into  continuous-time  (CT)  [Chua88bl  and  discrete-time  (DT)  [Rodr90,  Harr92]  models.  Each 
model  type  is  described  by  a  set  of  nonlinear  dynamic  equations,  one  per  cell,  who.se  associated 
equilibrium  state  distribution  determines  the  network  computational  properties.  Previously 
reported  CT-CNN IC  design  approaches  focused  on  the  u.se  of  g,n-C  techniques  for  the  implemen¬ 
tation  of  the  Chua- Yang’s  CNN  cell  circuit  model  [Chua88a].  Proposed  discrete-time  realizations 
focused  on  a  very  similar  cell  circuit  model,  for  which  MOSFET-C  techniques  and  fully  differential 
high-output  impedance  opamps  have  been  considered  [Harr92].  In  all  cases,  input  signals  are  volt¬ 
ages,  while  internal  signals  can  either  be  voltages  or  currents.  Since  primary  output  of  image  .sensor 
devices  (jdiototransistors  [Sayl91])  is  current,  the  need  arises  to  convert  these  outputs  to  voltage, 
thus  complicating  CNN  interface  design  for  image  processing  tasks.  Also,  electrical  cell  design  is 
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not  easy  because  different  dynamic  ranges  for  the  internal  voltages  and  currents  must  be  considered 
to  guarantee  a  reduced  influence  of  the  MOS  transistor  nonlinearities.  Finally,  operation  speed  is 
not  optimum  because  the  combination  of  internal  voltage  and  cuiTent  signals  results  in  internal 
high-impedance  nodes,  and  hence,  large  time  constants. 

Here,  a  new  CNN  cell  model  is  presented  and  implementation  techniques  are  discussed  for  both 
CT  and  DT  CNNs  using  current-mode  techniques.  Cell  complexity  is  shown  to  be  much  smaller 
than  for  previous  approaches.  Also,  design  is  very  simple  (only  two  sizing  equation  are  needed) 
and  the  speed/power  figures  are  very  good  due  to  the  lack  of  internal  high-impedance  nodes. 


Chua- Yang’s  CNN  ModelrAnaiog  Implementation. 

Fig.l  shows  the  Chua- Yang’s  CNN  cell  circuit  model,  whose  dynamic  is  given  by; 

=  -.v'  (0  +D‘  +  {A^it)  +By'}  Vc6  (J-D  (1) 

deNJc) 

where  A/^/cj  represents  the  cell  neif>hhorhood,  including  cell  c  itself,  Cj‘D  '\%  the  ntigrid  domain,  and 
tlie  cell  outputs  (y'^)  are  obtained  from  the  state  variables,  by  the  following  nonlinear  function, 

/(O  =  ^(k'(/)+l|-|.v''(/)-l|)  (2) 

The  input  {BJ  )  and  output  {Af)  weights  are  called  control  md  feedback  parameters,  respectively, 
and  D'  is  the  offset  parameter.  Computational  properties  of  Chua- Yang’s  model  rely  on  its  ability 
to  yield,  for  A'  >1,  two  stable  equilibrium  points  separated  by  an  instability  region,  and  in  the  pos¬ 
sibility  of  modifying  the  stable  point  attraction  regions  by  changing  the  neighbor  contributions, 

I  =  D‘+  X  {^>‘‘(0 (3) 

(/  6  N^(c) 

<1  *  c 

This  is  illustrated  in  .Fig.2,  showing  the  Xidjif  tdt)  vs  /  characteristic  for  the  different  qualitative 
situations  possible.  The  displayed  dynamic  routes  show  that  the  attraction  region  for  the  equilib¬ 
rium  point  on  the  right  becomes  narrower  for  /  decreasing;  for /=/j,  this  point  becomes  virtual.  A 
similar  situation  happens  for  the  equilibrium  point  on  the  left,  in  case  /  increases. 

Analog  VLSI  implementation  of  (1)  must  handle  different  variation  ranges  for  the  state  ai  d  the  out¬ 
put  variables,  as  illustrated  in  Fig.2:  while  the  output  variables  change  inside  [-1,1],  state  variable 
excursions  are  constrained  by  the  following  normalized  maximum  value  [ChuaSSa], 


.V 


nia.y 


+  |D'1+  I  iMil+M) 

tie  Npc) 


(4) 


which,  for  typical  templates,  ranges  between  5  and  10.  Thus,  in  the  previously  reported  g^-C  cir¬ 
cuits  [Cruz91,  Halo92],  where  all  resistive  components  in  Fig.l  are  realized  by  differential  input 
tran.sconductors,  largely  different  biasing  conditions  and  design  equations  must  be  considered  for 
the  transconductors,  thus  complicating  the  sizing  process  and  yielding  non-optimum  power  con¬ 
sumption  and  area  figures. 
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Extended  Range  CNN  Models 


A  new  CNN  model  which  preserves  qualitative  properties  of  the  Chua- Yang’s  model  and  yields 
identical  variation  ranges  for  both  state  and  output  variables  is  proposed  as  follows, 

=.?l-v  «)]+£)''+  y  Vrei;®  (5) 

deN.U) 

where  g(»)  is  a  three  pieces  piecewise  linear  characteristics  given  by, 

,-07  (jr‘^4- 1)  +  1 

g(.v')  =  lim  i  -y  otherwise  (6) 

-  777  -  1 )  -  1  >  1 

Convergence  towards  binary  states  in  the  new  model  is  illustrated  in  Fig.3.  It  is  seen  that  dynamic 
routes  converge  to  binary  states  for  all  possible  qualitatively  different  cases. 

A  similar  full  range  model  is  found  for  discrete  time  CNNs, 


/(/7+1)  =/ 


D' 


+  I: 


{/ly  (/I)  +By) 


Vce  tj'D 


which  involves  only  input  and  output  variables.  In  this  model,  convergence  towards  binary  output 
requires  that /(•)  be  sigmoidal  and  that  the  equivalent  slope  at  the  origin,  A^}  f(0),  be  larger  than  1 
to  ensure  regenerative  behavior.  This  can  be  achieved  by  using  either  a  soft  nonlinearity  such  as 
that  in  (2)  and  Aj.'  >l,  or  by  using  a  comparator^ 


m  = 


for 

for 


2>0 

2<0 


(8) 


and  A^/  =l,  as  proposed  in  [Harr92].  Convergence  towards  binary  outputs  for  the  DT-CNN  model 
is  illustrated  in  Fig.4. 


Current-Mode  CNNs  Conceptual  cells 

Signal  summation,  scaled  replication,  integration,  delay  and  nonlinear  transformation  are  the  ana¬ 
log  operators  required  to  implement  CNNs.  Summations  are  performed  in  current-mode  by  routing 
currents  to  a  common  node.  Remaining  operators  are  realized  using  a  very  simple  analog  building 
block:  the  current  mirror.  A  current  mirror  yields  linear  current  scaling  via  a  simple,  yet  ingenious, 
concept  involving  nonlinearity  cancellation  between  matched  transconductors.  Fig.5  shows  cur¬ 
rent-mirror  based  realizations  for  the  different  CNN  model  cells  given  above  using  a  generic, 
abstract  three  terminal  transconductor  (Fig.6)  such  that. 


h  =  Pu{v^,v^) 
i  1  «  /  2 

where  //(•)  is  assumed  invertible,  at  least  in  Vy,  the  characteristic  is  parameterized  by  a  designer- 
controlled  .scale  factor  P,  and  the  device  is  a.s.sumed  to  enter  a  cut-ojf  region,  with  /2=0,  when  input 
voltage  is  below  a  cut-in  value.  Fig.5(a)  applies  for  all  models  discussed  in  the  paper.  Switches 
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labelled  5,  and  S,  are  used  for  cell  initialization  puipose.  Fig. 5(b)  is  a  schematic  for  the  cell’s 
dynamic  part  in  the  Chua-Yang  model.  Corresponding  schematics  for  the  full  range  CT  model  is 
shown  in  Fig. 5(c).  Switch  R^.  in  both  schematics  is  used  for  initialization  purposes.  It  is  seen  that 
the  full  range  model  gives  simpler  circuits  than  the  Chua-Yang  model.  Fig. 5(d)  shows  the  schemat¬ 
ics  of  the  cell’s  dynamic  part  for  the  DT  CNN  model. 

In  the  simplest  case,  the  generic  transconductor  contains  only  one  transistor.  Besides,  only  one 
MOS  transistor  is  required  to  implement  an  analog  switch  with  zero  ON  offset  and  very  low  OFF 
leakage  (about  lOpA).  This  infers  that  for  this  simplest  case,  DT  CNN  cells  of  Fig.5  are  imple¬ 
mented  in  CMOS  using  only  18  tran.si.stors;  this  is  an  important  advantage  when  compared  to  pre¬ 
vious  approaches  for  DT  CNNs  [Harr92],  where  106  transistors  are  required  for  a  similar  cell.  For 
CT  Chua-Yang  CNNs,  complexity  (measured  in  number  of  transistors)  of  the  cuirent-mode  circuit 
IS  similar  to  previous  g^-C  implementations  [Cniz91J.  However,  this  complexity  decreases  in  cur¬ 
rent  mode  implementations  of  the  full  range  model  (down  to  18  transistors  per  cell).  Despite  the 
actual  model  considered,  the  design  equations  for  current  mode  CNN  ICs  are  much  simpler  than 
for  previously  reported  techniques  because  of  the  intrinsic  cument  min  or  nonlineai  ity  cancellation. 
Also,  cunent  mode  CNNs  are  faster  and  allow  larger  pixel  densities. 

CMOS  Current  Mode  CNN  Design  Issues 

Current  mode  CNN  operation  is  degraded  by  both  random  and  systematic  sources  of  enor.  Random 
errors  are  due  to  statistical  variations  of  the  technological  parameters  across  the  die  and  can  be 
attenuated  using  large  devices,  careful  layout,  and  proper  bias  generation  and  distribution.  System¬ 
atic  errors  are,  on  the  other  hand,  corrected  by  proper  tran.sconductor  choice  and  transistor  sizing. 
Two  major  systematic  error  sources  can  be  identified:  a)  Input-output  voltage  mismatching  at  the 
bias  point,  defined  as  the  point  where  transconductors  sink  only  their  bias  currents;  b)  Finite  RJ 
Rin  ratios,  where  R^  represents  tlte  tran.sconductor  output  resistance  and  /?,,,  holds  for  the  input 
resistance  (for  an  ideal  mirror,  RJRjn  should  be  infinitely  large). 

Static  Nonidealities  and  Sizing  Equations:  Voltage  mismatching  produces  current  offset,  origi¬ 
nated  by  the  transconductor  current  dependence  on  the  output  temiinal  voltage.  For  the  MOS 
transcoductors  in  the  paper,  this  offset  is  eliminated  by  forcing  the  same  current  density  in  the  mir¬ 
ror  input  ti  ansconductor  and  loading  devices.  Finite  RJRin  ratio  causes  current  gain  eiTor  due  to 
spurious  current  division  at  the  mirror  input  and  output  nodes.  The  en  or  is  especially  significant  if 
small  dimension  single-transistor  transconductors  are  used,  due  to  the  very  low  Early  voltages 
associated  to  .short  channel  transistors.  It  can  be  corrected  by  increasing  device  size  (channel 
lengtii)  but  this  does  not  yield  optimum  area  and  speed  for  CNN  implementations  in  scaled  down 
technologies.  For  improved  RjRj,,  figures  with  .short  channel  devices,  either  cascaded  transcon- 
diictors,  or  feedback  tran.scondiictors,  or  a  combination  of  both  mu.st  be  used  [Greg86,  Sack90]. 
Figs.7(a)  and  (b)  show  the  two  CMOS  mirror  structures  considered  for  CNN  design,  including  tlie 
complementary  devices  used  for  biasing  (dashed  lines).  We  have  chosen  n-channel  devices  for  the 
transconductors  since  their /f  values  are  larger  than  those  of  the  p-channel  (Kn>Kp),  yielding  more 
area-efficient  CNNs.  Desii’n  parameters  are  displayed  in  tlie  figures:  W,  L„,  and  Fig-7(c) 
sliows  a  circuit  to  provide  this  cascode  voltage.  We  assumed  that  Early  voltages  are  proportional 
to  the  channel  length:  V^,=a„L„  ,  V,^=a^^p.  In  Fig.7(a)  channel  lengths  are  different  for  NMOS 
(L„)  and  PMOS  (Z.^,),  to  obtain  equal  nominal  Early  voltages  for  both  devices,  and,  hence,  optimize 
R,JRj„.  In  the  case  of  Fig.7(b),  this  figure  is  intrinsically  much  larger  and  all  channel  lengths  are 
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made  equal  (L,j=L^,)  for  simpler  design.  To  achieve  simplicity,  we  have  also  assumed  that  all  tran¬ 
sistors  have  the  same  channel  width,  W.  Table  1  gives  sizing  equations  for  Figs.7(a)  and  (b).  These 
equations  are  intended  to  ensure  that  the  mirrors  handle  the  whole  input  current  range  with  mini¬ 
mum  distortion,  and  using  the  smallest  possible  devices.  The  W  expressions  given  in  the  table  cor¬ 
respond  to  a  bias  current  1q-,  W  values  for  larger  currents  (associated  eitlier  to  output 
transconductors  where  gain  is  larger  than  unity  or  to  input  tran.sconductors  with  bias  current  SIq) 
are  calculated  taking  into  account  tlie  requirement  for  equal  current  density  in  all  transconductors. 


Table  1:  Sizing  Equations  for  CMOS  Mirrors 


Channel  widths  (W=W„=Wi,) 

Lengths 

Bias  voltage 

single 

device 

mirror 

w  -  r /  “■  1’ 

a 

L  =  —L 
p  a  « 
p 

cascode 

mirror 

16/. 

W  -  ^  L 

K  Vr.. 

n  in 

■ 

Note  that  tlie  sizing  equations  are  parameterized  by  L„,  which  is  chosen  by  the  designer  to  control 
RJRj,,  and  the  channel  area.  Fig.8  shows  the  current  gain  error  per  cell  versus  the  total  cell  area  for 
a  full  range  CT  CCD  CNN  using  single  and  cascode  mirrors  in  n-well  1 .6)am  CMOS;  the  top  family 
is  for  the  single  transistor  mirror,  and  the  bottom  family  for  the  cascode.  Parameter  is  the  rail  cur¬ 
rent  Iq,  which  vaiies  from  0.25pA  to  128nA.  As  it  can  be  seen,  simple  current  mirror  requires  large 
area  to  achieve  acceptable  error  figure.  On  the  other  hand,  cascode  mirrors  allow  using  short  chan¬ 
nel-length  devices,  and,  thus  result  in  much  higher  area  efficiency. 

Dynamic  Nonidealities;  It  can  be  seen  that  the  time  constant  of  the  CT  models  depends  on  the  state 
variable  value;  for  instance,  for  the  Chua-Yang  model  tlie  following  is  obtained. 


C 


(10) 


Thus,  transient  is  faster  if  the  ceil  state  is  near  the  black  pixel.  However,  the  equilibrium  points  are 
exactly  the  same  as  for  the  nominal  ca.se,  corresponding  to  the  solution  of  the  equation. 


If  =  -.r'+^(|A-‘(0+/ 


A‘'(0-/d)+/  =  0 


(11) 


Also,  the  sign  of  tlie  state  variable  derivative  (equivalently,  the  sign  of  If  )  for  a  given  /  has  exactly 
the  saine  dependence  on  a'  as  for  the  nominal  ca.se,  and,  as  a  consequence,  actual  dynamic  routes 
are  identical  to  the  nominal. 

Another  dynamic  error  arises  in  the  analog  switches  due  to  the  necessity  to  evacuate  the  MOS 
channel  charge  during  the  switch  turn-off  transient  process.  This  error,  generically  called 
feedthroui>h  error  [Eich89],  is  very  large  (up  to  20%  and  more)  if  small  geometries  transconductors 
are  used.  A  simple  technique  to  attenuate  this  error  is  the  inclusion  of  an  additional  capacitor  at  the 
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switch  output  node.  Since  neither  linearity  nor  accuracy  in  the  capacitance  is  required  for  this  pur¬ 
pose,  a  sliorted  transistor  can  be  used.  This  is  feasible  for  standard  digital  CMOS  technologies 
(having  only  one  poly  layer)  and  require  less  area  than  typical  capacitors  in  two  poly  technologies 
(Polyl-Si02-Poly2),  since  channel  oxide  is  usually  tliinner  than  interpoly  oxide.  The  use  of  tliis 
technique  can  easily  lower  the  error  about  one  order  of  magnitude.  Also,  since  only  two  of  these 
devices  are  required  in  each  CNN  cell,  area  penalization  is  not  severe.  Much  lower  feedthrough 
enor  is  achieved  using  a  dummy  transistor,  and  automatically  tuning  the  delay  between  the  switch¬ 
ing  device  clock  signal  and  the  dummy  device  clock  signal  [Espe92].  In  tliis  manner  errors  as  little 
as  0.3%  have  been  measured  on  silicon  prototypes.  Also,  the  extra  monitoring  and  control  circuitry 
required  for  this  automatic  tuning  can  be  shared  for  all  cells  in  the  network,  so  that  area  penalty  is 
not  severe  at  the  network  level. 

Bias  Current  Selection:  A  crucial  issue  that  has  not  been  discussed  yet  is  the  election  of  the  bias 
current  !q.  The  geometry  factor  {WIL)  of  the  transi.stors  in  Figs.7(a)  and  (b)  increases  monotoni- 
cally  with  Iq.  Furthermore,  static  gain  error  due  to  finite  R,)/Ri„  values  also  increases  with  Iq.  In 
addition,  power  dissipation  is  proportional  to  the  bias  cuirent.  For  these  reasons,  a  bias  current  as 
small  as  possible  should  be  choosen.  The  issue  is  to  identify  the  minimum  feasible  rail  current 
value.  Lowest  limit  of  the  bias  current  is  certainly  established  by  leakage  (about  lOpA  in  standard 
CMOS).  However,  a  more  restrictive  bound  exists  due  to  MOS  transistor  mismatch  [Pelg89]  and 
Early  voltage  (V^)  degradation  with  channel  length.  The.se  phenomena,  which  actually  limit  the 
minimum  devices  geometries,  also  restrict  the  minimum  current. 

Mismatch  is  mainly  produced  by  variations  of  the  threshold  voltages  (Vy)  and  large  signal 
transconductance  (p=fiC<,^.W/L)  of  equally  designed  transi.stors  in  the  same  chip.  Standard  devia¬ 
tion  aiVi)  and  ratio  a(P)/P  can  be  expressed  in  terms  of  two  stati.stically  different  contributions:  a 
white  espectra  component,  that  accounts  for  the  effect  of  phenomena  with  extremely  short  corre¬ 
lation  distance  (much  less  tlian  minimum  transistor  dimensions),  and  a  second  component  that 
reflects  the  effect  of  large  correlation  distance  phenomena.  Tlie  first  component  is,  in  a  good  aprox- 
imation,  inversely  proportional  with  the  square  root  of  the  channel  area,  while  the  second  is  pro¬ 
portional  to  the  distance  between  two  equally  layed-out  devices.  Results  in  [Pelg89]  demonstrate 
that  the  distance  dependent  component  is  negligible  for  devices  with  a  channel  area  less  than  about 
100|J.m“.  For  bias  cunents  below  about  50p.A  and  ca.scode  minors,  design  equations  in  Table  1  give 
devices  with  channel  area  well  below  this  bound.  Hence,  the  distance  dependent  component  need 
not  to  be  considered  for  current  mode  CNNs,  but  only  the  c{Vi)  and  a(P)/p  dependence  with  chan¬ 
nel  area. 

In  adition,  for  a  given  a(Vy)  and  a(P)/p,  the  ratio  o(/)//  in  MOS  tiansistors  is  inversely  proportional 
to  the  gate-source  voltage  Vj,^..  This  means  that,  once  W/L  factors  have  been  set  to  achieve  accept¬ 
able  mismatch  levels,  current  bias  can  not  be  decrea.sed  too  far  below  the  upper  bound  given  by 
equations  in  Table  1,  since  this  would  produce  a  low  voltage  in  the  bias  point,  with  the  corre¬ 
sponding  high  a(/)//.  Hence,  mismatch  considerations  e.stablish  bounds  for  both  minimum  area  and 
power  trends.  For  example,  we  have  obtained  100%  success  (out  of  30  trials)  for  a  Montecarlo  sim¬ 
ulation  of  a  connected  component  detector  (CCD)  current  mode  full  range  CT  CNN  with  16  cells 
in  a  row  using  unitary  transistor  geometries  of  VV'/L=4}im/3.2|im  for  both  n  and  p-channel  devices, 
and  a  bias  current  lQ=2[iA.  Further  geometry  reduction  has  not  been  considered,  since  minimum 
contact  size  (4pm  with  surrounding  diffusion  in  the  1.6pm  n-well  technology  used)  does  not  allow 
a  significant  area  reduction  anyway.  Similar  simulations  performed  on  a  Chua-Yang  model  coun¬ 
terpart  resulted  in  lower  yield  figures,  which  can  be  understood  by  observing  that  the  normalized 
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vaiiie  dispersion  of  the  equilibrium  points  (normalization  factor  is  the  distance  between  nominal 
equililibrium  state  positions)  is  much  smaller  for  the  full  range  model,  than  for  the  Chua-Yang 
model.  Hence,  larger  geometries  should  be  used  for  increased  yield  with  this  model. 

Current  references  for  eveiy  cell  in  the  network  and  for  every  current  source  within  each  cell  can 
be  generated  from  common  bias  voltages.  Reference  di.spersion  due  to  mismatch  among  tlie  tran¬ 
sistors  of  different  cuirent  sources  did  not  produce  critical  results  in  Montecarlo  simulations.  Cas- 
code  voltage  for  cuirent  rriirrors  can  also  be  globally  distributed.  If  high  noise  levels  are  expected 
at  bias  voltages,  as  may  be  the  case  in  discrete  time  implementations,  a  bias  voltage  independent 
current  reference  [Greg86],  shown  in  Fig.9,  can  be  included  in  each  cell,  although  a  strong  area 
penalization  may  result.  Cuirent  reference  dispersion  will  be  produced  only  by  local  mobility  and 
resistivity  variations.  Montecarlo  simulation  of  entire  CCD  systems  where  cell  cunent  references 
have  a  standard  deviation  larger  than  5%  have  shown  100%  success  (out  of  15  trials). 

Discussion  of  Results 

Table  2  gives  the  transistor  count  and  total  cell  area  for  different  templates,  and  for  both  CT  models. 
Cascode  mirrors  with  1^=411111  and  C=3.2|im  are  used.  For  DT  implementations  area  is  slightly 
larger  than  those  for  the  full  range  CT  model,  due  to  the  switches.  Although  data  in  Table  2  do  not 
include  the  area  occupied  by  the  initialization  circuitry,  pixel-densities  ranging  from  60  to  more 
than  160  cells/mm  can  be  easily  achieved  (depending  on  the  particular  template)  if  the  full  range 
model  is  u.sed. 


Table  2:  Cell  area  and  transistor  count  for  different  templates  and  CNN  models. 


Full  Dynamic 
Range 

Area  (lim^) 

Chua-Yang 

Model 

Area  (|am^) 

Full  Dynamic 
Range 

Ttof.  Count 

Chua-Yang 

Model 

Ttor.  Count 

C.  C.  Detec. 

5916 

12691 

40 

56 

Shadow  Detec. 

7736 

16533 

52 

68 

Borders 

Extrac. 

15471 

26291 

112 

128 

Corners 

Extrac. 

16381 

28212 

120 

136 

Hole  Filling 

10921 

24774 

76 

92 

Noise  Filtering 

5460 

14258 

40 

56 

Several  l.Opm  CMOS  prototypes  have  been  designed  to  probe  ideas  in  the  paper:  one  is  DT  and 
can  be  reconfigured  via  local  logic  for  different  templates,  other  two  prototypes  are  CT  and  fixed 
template  (CCD  and  other  for  noise  removal).  Obtained  results  demonstrate  the  possibility  to 
achieve  very  large  cell  densities  with  good  yield  figures  by  using: 

*Full  range  CNN  models,  allowing  all  cell  variables  to  have  the  .same  variation  ranges  and, 
hence,  better  area  and  power  consumption  figures. 

^Intrinsic  functional  nonlinearity  cancellation,  giving  technology  independent  circuits  and 
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ai  chitectures,  and  allowing  the  simplification  of  the  IC  design  process. 

*Cascode  transistors,  to  reduce  .static  error  ternis  without  area  and  speed  penalization. 
*Cai‘eful  consideration  of  electrical  issues  related  to  net  architecture  and  input/ouput  interfac¬ 
ing. 

For  CMOS  technology,  current  as  low  as  2fiA  can  be  u.sed  with  reasonable  yield,  if  .strong  inversion 
is  used.  Speed  for  these  low  current  levels  is  optimum  (about  l|is  for  16  cells  CCD)  due  to  the  small 
device  dimensions  and  the  low  impedance  of  internal  nodes.  Electrical  tunability  can  be  easily 
incorporated  using  electrically  parameterized  transconductors,  for  instance,  differential  amplifiers. 
Digital  tunability  is  direct  by  switching  lines  rooted  to  comtnon  nodes.  Delay  templates  are  in  this 
way  easily  incorporated. 
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Abstract 

The  design  of  a  CMOS  analogically  programmable  cellular  neural  network  is  reported.  The 
electrical  characteristics  of  the  basic  building  blocks  are  analysed  and  discussed. 
Additionally,  some  performances  of  a  10x10  CNN  are  reported. 

1  INTRODUCTION 

In  recent  years  a  new  and  alternative  approach  to  the  classic  neural  network  topologies, 
called  Cellular  Neural  Networks  (CNN’s),  was  introduced  by  Chua  et  al  [1,  2].  CNNs 
exhibit  several  interesting  features  and  properties:  each  cell  is  locally  connected  only  to  its 
neighbours;  the  processed  signals  are  analog  in  nature;  the  circuit  architecture  is  space 
invariant  and  this  renders  the  CNNs  particularly  suited  for  an  implementation  in  a  VLSI 
technology,  like  CMOS; 

Up  to  now,  only  a  few  proposals  of  practical  CNN  implementations  have  been  published  [3 
-  5]  and,  as  a  matter  of  fact,  the  8x8  connected-component  detector  (CCD)  of  [5]  represents 
the  only  CNN  chip  fabricated  and  tested  so  far. 

In  practice,  the  possibility  to  manage  a  neural  network  which  can  be  used  for  different 
applications  in  different  time  periods  of  the  signal  processing  procedure  is  particularly 
interesting,  as  one  can  use  the  same  part  of  the  chip  to  perform  several  circuit  functions.  In 
this  context,  a  CMOS  approach  for  the  design  of  a  reconfigurable  cellular  network  in  which 
the  template  coefficients  can  be  digitally  varied  has  been  proposed  in  [4]. 

In  this  work  we  will  present  the  design  of  an  analog  programmable  CNN  architecture  with 
low-power  dissipation  in  a  1.5um  CMOS  technology. 

In  particular,  after  discussing  the  design  of  basic  building  blocks,  we  will  report  the 
electrical  performances  of  a  10x10  CMOS  CNN,  constituted  of  about  8,000  MOS 
transistors,  fully  simulated  at  the  device  level,  which  can  L.  mply  analog-programmed  by 
varying  an  external  control  voltage.  By  so  doing,  the  designed  CNN  can  perform  such 
function  as  noise  removal,  hole  filler,  shadow  detector,  connected  component  recognition 
and  edge  detector.  The  whole  power  consumption  of  the  circuit  is  limited  to  about  60mW, 
which  is  about  1/3  compared  to  the  power  consumption  of  the  non  programmable  circuit 
presented  in  [3]. 

A  programmable  CNN  architecture  can  be  used  for  several  applications  in  image  processing. 
For  example,  hand-written  character  recognition  is  known  to  be  very  sensitive  to 
preprocessing,  which  differs  for  constrained  and  unconstrained  writing,  moreover,  for  the 
single  classifiers  on  cooperating  recognition  systems.  A  programmable  CNN  architecture 
used  for  the  normalization  stage  of  the  classifier  allows  to  apply  the  same  hardware  to  the 
above  cases.  The  image  size  is  usually  32x32  or  less  and  requires  the  processing  of 
200+2000  characters  per  second,  making  the  CNN  architecture  a  very  good  candidate  for 
this  purpose. 
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2  DESIGN  OF  THE  CMOS  BASIC  BUILDING  BLOCKS  FOR  THE 
CNN 

The  processing  unit  (N-shaped  resistor)  was  achieved  by  feedback  connecting  an  inverting 
transconductance  CMOS  operational  amplifier.  The  schematic  diagram  of  the  circuit  is 
shown  in  Fig.  1,  while  Fig.  2  reports  the  simulated  I(V)  characteristics  of  the  designed 
circuit.  As  it  can  be  seen,  the  current  and  voltage  values  corresponding  to  the  two  knee 
points  in  the  curve  are  symmetric  and  their  value  are  luA  and  0.6V,  respectively. 

The  linear  voltage-dependent  current  sources  required  to  realize  the  coefficients  for  template 
B  were  realized  with  the  simple  OTA  circuit  shown  in  Fig.  3,  which  has  the  inverting  input 
always  connected  to  ground.  Fig.  4  reports  the  simulated  characteristic  of  the  circuit  for  a 
design  chosen  to  give  a  weight  of  2.  For  a  ±5V  bias  voltage  the  two  above  circuits  have  a 
power  consumption  of  about  75uW  and  55uW,  respectively. 

Tlie  circuit  used  for  analog  programming  the  coefficients  for  feedback  template  A  is  shown 
in  Fig.  5  and  was  developed  with  the  aim  of  reducing  at  most  the  number  of  control  lines, 
the  number  of  MOS  devices,  the  active  gate  area  and  the  power  consumption.  It  is 
constituted  by  a  6  transistor  multiplier  cell,  a  current  mirror  M1-M2  which  establishes  the 
current  of  the  main  circuit,  a  simple  output  mirror  M9,  Ml 4  and  a  double  output  current 
mirror  M10-M13. 

In  particular,  the  drain  current  Id2  of  device  M2  is  selectively  diverted  trough  M3  or  M4 
depending  on  the  difference  between  the  input  control  voltage  Vc  and  the  reference  voltage 
Vbias.  The  partition  of  Id2  trough  M3  and  M4  constitutes  the  programming  action  of  the 
control  voltage  Vc.  The  circuit  enables  to  get  for  the  lateral  coefficients  of  the  template  A 
values  which  can  be  continuously  varied  from  8  down  to  -8. 

Fig.  6a  shows  a  typical  behaviour  of  the  saturation  output  current  los  as  a  function  of  Vc, 
while  Fig.  6b  shows  the  output  current  lo  as  a  function  of  the  state  voltage  Vx  of  the  cell, 
for  several  values  of  A  ranging  from  -0.25  to  -8. 

3  APPLICATIONS 

By  using  the  building  blocks  previously  described  we  designed  several  10x10  CNNs  with 
the  aim  of  validating  the  design  of  the  programmable  A  generator.  The  parasitics  effects 
as.sociated  to  overlap  capacitances  and  junction  capacitances,  which  vary  by  changing 
transistor  dimensions  and  bias  voltages,  were  accurately  taken  into  account.  The  resulting 
circuits  have  a  complexity  of  the  order  of  8000  transistors  and  were  completely  simulated  at 
the  device  level  by  using  the  circuit  analysis  program  Spice.  The  state  capacitor  was  IpF. 

3.1  CNN  Low  Pass  Filter  for  Noise  Removal 

In  this  application  (LP-CNN)  the  image  was  continuously  applied  to  the  input  and 
applied  as  initial  condition  to  the  state  capacitors.  The  feedforward  template  B  and  the 
"first  choice"  for  the  feedback  template  A  were  the  same  of  Ref.  [3]. 

The  designed  LP-CNN  was  tested  by  processing  several  input  images  with  both  different 
voltage  levels  corresponding  to  the  black  and  white  colours  and  different  superimposed 
gaussian  noise.  For  black  and  white  levels  shrinked  together  down  to  60%  of  the 
dynamic  range  and  a  gaussian  noise  with  a  standard  deviations  not  higher  than  0.225  the 
designed  LP-CNN  perfectly  removes  the  input  noise.  Conversely,  when  the  dynamic 
range  of  the  input  signal  was  made  smaller  (40%  of  the  dynamic  range)  and  the  standard 
deviation  of  the  noise  was  increased  (s  =  0.75)  the  LP-CNN  shows  errors  in  noi.se 
removal. 
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For  an  image  with  s  =  0.75  an  example  is  shown  in  Fig.  7,  where  the  output  image  found 
by  the  LP-CNN  after  noise  removal  (b)  and  the  correct  image  (a)  are  reported.  There  are 
7  pixels  containing  an  error  in  the  output  image  (a  similar  result  was  achieved  in  [3]). 
The  time  required  from  the  LP-CNN  to  reach  the  equilibrium  state  is  less  than  2usec. 
Due  to  the  programmability  of  the  Aij  coefficients,  several  other  analyses  were 
performed  by  continuously  varying  the  value  of  the  "lateral"  weights  in  the  A  matrix 
(which  were  1  in  the  original  case)  from  1.5  down  to  0.  We  found  that  by  increasing  Aij 
the  final  values  of  the  state  voltage  slightly  change,  but  the  number  of  errors  Ne  in  the 
output  image  does  not  change.  Conversely,  when  the  values  of  the  lateral  .Aij’s  decrease 
gradually  from  1.0  down  to  0.25,  we  found  more  pronounced  variations  in  the  final 
values  of  the  state  output  voltages  and  also  a  sensible  decrease  in  the  number  of  errors. 
The  above  small  sensitivity  of  the  output  image  to  variations  in  the  values  of  the  lateral 
Aij  weights  seems  indicate  that:  a)  the  "preprocessing"  filtering  action  given  by  the 
template  B,  which  is  permanently  applied  on  the  input  image,  is  very  strong  and,  b),  the 
template  A  seems  just  to  establish  the  transient  behaviour  toward  well  defined  final  state 
voltages. 

These  observations  were  further  validated  from  the  results  achieved  by  programming  the 
Aij’s  equal  to  zero:  in  such  a  case  when  the  LP-CNN  just  does  a  dynamic  filtering  action 
in  which  the  state  of  each  cell  is  independent  on  the  evolution  of  the  state  voltage  of  the 
neighbouring  cells,  we  do  not  found  a  variation  in  Ne. 

3.2  CNN  for  Edge  detection 

For  this  application  we  chose  for  the  templates  A  and  B  the  same  values  reported  on  Fig. 
15  of  [2].  As  a  processing  example  we  used  the  10x10  diamond  structure  of  Fig.  8a.  The 
result  achieved  by  programming  the  template  coefficient  values  are  shown  in  Fig.  8b.  As 
can  be  seen,  results  equal  to  those  found  by  using  the  theoretical  model  of  [1]  have  been 
obtained.  Additionally,  results  equal  to  those  found  in  [2]  by  varying  the  template 
coefficients  were  found. 

3.3  Other  examples 

The  designed  CNN  was  also  programmed  to  operate  other  functions  useful  for  image 
processing  such  [6  -  9]  as  hole  filler,  connected-component  detector  and  shadow 
detector.  In  all  the  examined  cases  the  circuit  simulation  indicates  a  correct  operation  of 
the  CNN. 
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Figure  7a 


Figure  7b 


Figure  8b 
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Abstract 

An  operational  transconductance  amplifier  with  multiple  inputs,  each  of  them 
realized  by  means  of  two  MOSFETs  only,  suitable  for  implementation  of 
Cellular  Neural  Networks  in  CMOS  technology  is  described.  Preliminary 
analysis  of  s ho r t -channe  1  circuit  augmented  by  SPICE  simulation  is  given. 
Results  show  potential  ability  of  designing  chip  area  effective  networks 
applicable  to  various  tasks  of  image  processing  and  pattern  recognition  in 
real-time.  This  needs  extention  of  the  research  especially  in  the  aspect  of 
VLSI  implementation. 


1.  Introduction 

Even  though  the  Cellular  Neural  Networks  (CNN)  as  originally  proposed  in 
[1,2]  are  expected  to  be  very  efficient  in  VLSI  implementation  the  research 
on  this  subject  has  not  succeded  so  far  in  establishing  standard  circuit 
realization  of  a  single  cell  and  architecture  of  the  network  as  a  whole. 
Structures  of  the  cell  circuits  proposed  in  [3-6]  are  still  far  from  solution 
which  can  satisfy  CNN  requirements  and  more  study  augmented  by  computer 
simulation  and  verification  with  VLSI  test  chips  are  highly  recommended.  The 
first  succesfully  constructed  chip  of  CNN  connected  component  detector  [7]  is 
very  encouraging  sign  and  stimulates  the  research  in  the  hope  of  achieving 
significant  progress  in  this  area. 

Main  question  in  the  design  of  CNN  cell  circuit  in  VLSI  technology  is 
how  to  solve  efficiently  the  problem  of  controlling  the  cell  by  signals 
coupled  in  from  itself  and  other  cells  of  given  neighborhood  in  the  network. 
In  this  sense  efficiently  means  realizability  taking  into  account 
restrictions  of  the  VLSI  design  rules,  minimization  of  the  chip  area  and 
power  consumpticn  as  well  as  flexibility  in  implementation  of  dynamical 
processing  algorithms  suitable  for  specific  applications  of  CNNs. 

List  of  design  requirements  for  VLSI  CNNs  includes  the  following 
technological  problems: 

1.  Realizability  of  feedback  operator  A  (cloning  template),  control 
(feedforward)  operator  B  and  polarization  vector  I.  This  requirement  is 
limited  by  wiring  method  and  floorplanning  of  the  VLS  layout.  Due  to  this 
limitations  not  every  CNN  can  be  realized  in  a  given  technology. 

2.  Programmability  of  operators  A,  B  and  I,  defined  as  the  ability  to  alter 
the  values  of  the  weights  between  cells  in  a  given  neighborhood  and  values  of 
the  polarization  vector  I.  This  ability  extends  CNN  tasks  that  can  be 
realized  in  image  processing  and  pattern  recognition. 

3.  Method  of  inputing  to  and  outputing  from  the  CNN  the  visual  information  to 
be  transformed. 
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4,  Compatibility  with  digital  raicroprocesor  systems  which  enables 
connection  of  CNNs  as  a  extending  cards  for  speeding  up  calculations. 

5.  An  influence  of  nonidealities  and  tolerances  of  semiconductor  elements  as 
well  as  frequency-dependent  loss,  crosstalk  and  immunity  to  electromagnetic 
interference  for  stability  of  the  CNNs. 

Taking  into  account  the  main  requirements  as  stated  above  we  present 
main  results  of  our  preliminary  study  of  potential  applications  the  concept 
of  multiple-input  OTA  circuit  in  CMOS  technology  [8].  We  concentrate  only  on 
the  problem  of  designing  the  main  block  of  CNN  cell  circuit,  namely  the  block 
of  spatial  convolution  the  outputs  from  neighborhood  cells.  Referring  to  the 
nonlinear  differential  equation  [2]: 

c  V  -  V  V  R  +  A'^  V  '*  f  V  ■'  +  r  fl) 

Xx  XX  cly  du 

which  describes  the  dynamics  of  the  cell,  this  block  realizes  summation  of 
the  three  last  components  of  the  rigth-hand  side  of  (1)  or  equivalently  the 
summation  at  node  N  currents  comming  from  voltage-controlled  current  sources, 
Fig.l. 


Results  obtained  during  this  work  empower  us  to  develop  this  idea  much  more 
extensively,  especially  in  the  aspect  of  optimization  of  the  VLSI  cell  and 
programmability  of  the  network  in  real-time  image  processing  systems. 


2.  Multiple-input  OTA  circuit  for  CNN  cell 

The  fundamental  circuit  of  multiple-input  OTA  as  proposed  in  [8]  is 
shown  in  Fig. 2.  The  OTA  is  driven  by  two-transistors  input  structures  (M^^^  - 

M^y)  connected  in  parallel  to  both  sides  of  current  mirror  M^  -  M^.  The 

circuit  output  current  is  equal  to  the  difference  in  node  currents  and 
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as  its  bias  current 

Computer  simulation  shows  that  the  error  introduced  in  the  approximation 
(2)  is  negligible  when  the  aspect  ratio  (W/L)^^  is  at  least  200  times  greater 

then  the  aspect  ratio  (W/L)^.  Assuming  2  pm  CMOS  feature  size  this  creates 

large  chip  area  occupied  by  single  input  and  correspondingly  very  large  area 
by  the  whole  cell  (one  cell  can  be  controlled  by  16  current  sources  even  in 
the  case  of  minimal  neighborhood  radius). 

To  safe  chip  area  the  MOSFET  technology  with  channel  length  down  to  1  pm 
or  even  smaller  should  be  used.  However,  for  such  small  channel  length  new 
transistor  model  with  carriers  drift  velocity  effect  must  be  considered. 
The  use  of  the  long-channel  square-law  model  for  the  short-channel  devices 
gives  considerable  discrepancies  in  both  the  dc  and  transient  solutions 
between  the  calculated  and  measured  results  and  can  lead  to  incorrect 
conclusions  in  regard  to  circuit  performance.  It  was  shown  [9],  that  when  the 
power  supply  of  +5V  and  -5V  is  used  (which  is  typical  standard  supply  in 
digital-analog  integrated  circuits)  then  the  MOS  transistor  model  in 
saturation  for  short-channel  device  can  be  described  by; 


I 
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(3) 


where  is  carrier  drift  velocity  saturation  equal  to  approximately  7xl0'' 

cm/s  for  both  the  NMOS  and  PMOS  devices.  As  was  expected  in  the  theory  of 
short-channel  devices,  the  drain  current  in  saturation  region  of  I-V 
output  characteristics  does  not  depend  on  the  channel  length.  Also,  the  bulk 
effect  is  negligible  and  the  output  conductance  of  the  device  is  included 
through  the  threshold  voltage  dependent  on  drain-source  voltage. 


The  I-V  characteristic  of  the  k-th  two  transistor  structure  (Fig. 3)  with 
short-channel  devices  can  be  derived  by  equating  the  drain  current  of  the 
lower  transistor  M^^^  operating  in  nonsaturation  with  the  drain  current  of  the 

upper  transistor  operating  in  saturation  region.  After  all  stages  of  this 

derivation  one  can  obtain  the  following  formula  for  the  output  current  I^^: 

K  =  [  V.  -V,-  0.5  (  V  -V  )  ]  [1  +  a(V  -V  )]  (4) 

k  kbkT  ikT  bkT  ikT 

where  the  constant  a  is  expressed  by  (L^^  is  the  channel  length  of  the  lower 
transistor): 

a  =  p  W  /(V  W  L  )  (5) 

o  L  sat  U  k 

Assuming  transistors  geometry;  (W/L)^  =  (lOpm/lpm)  and  (W/L)^  =  (Ipm/lOpm) 

and  the  value  of  electrons  mobility  p  =  500  cm  /V.s,  constant  a  gets  the 

value  a  =  0.007  V  .  Therefore,  for  the  voltage  range  -5V  to  +5V  the 
following  enequality  holds:  1  a(V.^-V^).  In  this  case  the  formula  (4)  for 

the  output  current  of  the  transconductance  element  with  short-channel  devices 
is  equal  to  the  corresponding  formula  (2)  valid  for  long-channel  devices,  but 
with  considerable  reducing  of  atrea  uccupied  by  the  element. 

Fig. 3  shows  dc  characteristics  of  the  transconductance  element.  It  can 
be  noted  that  for  V.  in  the  range  between  +5  and  +10V  the  element  behaves 

like  a  linear  transconductor  with  the  error  of  maximum  value  not  exceeding 
20%  .  Since  neural  networks  do  not  require  a  very  linear  response,  this  error 
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can  be  quite  well  acceptable  for  many  CNN  applications. 


CCL) 


Fig. 3.  Two-transistor  transconductance  element  with  short-channel  devices 

(a)  and  its  DC  transfer  characteristics 


Fig. 4.  (a)  -  dc  voltage  transfer  characteristics,  (b)  -  dc  current  transfer 

characteristics  for  two-inputs  OTA  circuit  of  Fig. 2.  Results  were  obtained 
using  SPICE  simulation  with  MOSFET  model  Level  1 


Fig.4.  shows  simulation  results  of  DC  voltage  and  current  transfer 
characteristic  of  two-input  OTA.  It  is  seen  from  this  figure  that  the  linear 
range  of  the  output  voltage  spans  of  several  volts  while  controlling  the 
input  almost  from  minimal  to  maximal  voltage  of  power  supply.  This  output 
voltage  range  is  also  quite  well  acceptable  for  most  applications  of  CNNs. 
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3.  Conclusions 


A  multiple-input  OTA  circuit  as  proposed  in  [8]  has  been  adopted  in 
preliminary  analysis  of  VLSI  CMOS  implementation  of  Cellular  Neural  Networks. 
The  circuit  generates  an  output  current  which  is  a  linear  function  of  sum  of 
the  feedback  and  feedforward  signals  coming  from  neighbour  cells. 

The  circuit  enables  to  alter  the  transconductance  value  of  each 

voltage-controlled  current  source  by  means  of  changing  the  bias  voltage 

of  upper  MOSFET  device,  as  well  as  by  means  of  changing  the  aspect  ratio 

(channel  width  to  channel  length)  of  the  lower  (controlled)  device.  This 
makes  good  flexibility  in  the  design  of  CNN  cells  and.  the  whole  circuit 
for  given  image  processing  and/or  pattern  recognition  applications.  SPICE 
simulation  and  hand  analysis  of  the  circuit  with  short-channel  devices  show 
quite  good  linearity  with  eccnomical  occupation  of  chip  area.  Future  work 
should  be  concentrated  on  simulations  dynamical  characteristics  and  on  design 
procedure  for  specific  CNN  applications. 
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Abstract 

In  [5]  eui  analog  implementation  of  discrete-time  cellular  neural  networks  is  described  that 
is  based  on  the  idea  of  conductance  multipliers  [1],  [10],  [7]  and  transconductance  amplifiers 
[8],  [11].  The  circuit  architecture  takes  advantage  of  differential  currents  suppressing  dc  dis¬ 
turbances.  The  paper  gives  new  measurement  results  that  compare  the  fabrication  tolerances 
between  different  cells  on  the  chip  and  the  offset  of  the  single  cells. 


1  Introduction 


Cellular  Neural  Networks  [2]  have  been  shown  to  be  an  efficient  tool  in  image  processing.  How¬ 
ever,  the  whcde  capability  of  the  system  is  only  exploited  if  specific  hardware  realizations  are 
provided.  Analog  VLSI  implementations  have  the  advantage  that  the  CMOS  transistors  are  not 
only  used  as  switches  leading  to  a  very  small  number  of  devices  and  therefore  low  chip  area.  On 
the  other  hamd  analog  circuits  are  much  more  sensitive  to  disturbances  and  can  only  be  applied 
if  the  architecture  is  robust  enough.  For  cellular  neural  networks  analog  proposals  are  given  in 
[4],  [3].  In  [9]  a  circuit  is  described  realidng  a  discrete-time  implementation  of  this  architecture. 
The  circuits  differ  in  programmable  and  hxed  template  coefficients  or  in  an  additional  local  lo^c 
for  the  output  states. 

Discrete-time  cellular  neural  networks  (DTCNN)  [6]  are  derived  from  CNN  and  the  discrete 
Hopfield  model.  The  network  is  described  by  the  recursive  algorithm 


*■’(*)=  E  E 

d€JV,(c)  <1€N.(c) 

^(k) = /(x«(*  -!))={  ^ » 

if  time-invariant  templates  and  inputs  are  assumed.  The  variables  and  coefficients  denote: 


x^(k):  cell  state  o^:  feedback  coefficient  Nr{c):  r-neighbourhood 

y^{k):  cell  output  6^:  contrd  coeffident 
cell  input  threshold 


0-7803-0875-1/92  3$.00  ©1992IEEE 


163 


The  cells  are  only  locally  connected  within  the  r-neighbourhood  [2],  which  is  defined  as  the  set 
of  all  cells  within  the  distance  r  including  cell  c.  In  contrast  to  the  CNN,  the  nonlineaxity  is  a 
threshold  function  and  the  system  is  clocked.  The  constant  part  can  be  summarized  to  the  cell 
bias 

‘‘  =  fc»=  E  tV  +  f-  (3) 

d€Nr(c) 

2  Circuit  Architecture 


An  analog  circuit  is  discussed  in  [5],  [6]  for  realizing  the  DTCNN.  It  implements  the  feedback 
part  with  a  1- neighbourhood  on  a  hexagonal  grid  and  programmable  weights.  Fig.  1  gives  the 
circuit  structure  for  a  single  cell  realizing  the  algorithm  (1),  (2). 


Figure  1;  Feedback  part  realization  of  a  single  cell. 

The  operational  transconductance  amplifier  OTA2  is  used  within  the  linear  range  to  provide  a 
differential  output  current  that  is  proportional  to  the  input  voltage  w?,  which  corresponds  to 
the  constant  cell  bias  k‘^.  The  capacitor  C3  realizes  an  analog  memory  and  is  charged  via  the 
bus  IN,  which  is  activated  by  SIl.  This  enables  data  transfer  by  the  matrix  concept,  where  a 
whole  column  is  read  in  parallel.  The  feedback  of  the  binary  outputs  v'y  =  +  Vj  is  achieved 

by  the  conductance  multipliers,  which  are  controlled  by  the  global  weight  voltages  ±v^.  Here, 
Vj  denotes  the  threshold  voltage.  Their  output  currents  are  summed  and  transformed  into  a 
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voltage  and  ^r2-  Then  a  difference  voltage 


v^ikT)  =  v^.ikT)  -  v^,{kT)  =  R^ii^kT)  -  i^^kT))  =  i2x  E  ((*  "  1)^’)  + 

\d€Nr{c) 

is  obtained  ,  where  gm  denotes  the  conductance  of  the  OTA2  and 

W 

At  =  2—tiCo.vf. 


(4) 

(5) 


The  parameters  W  and  L  define  the  transistor  geometry;  n  is  the  mobility  and  Cox  the  oxide 
capacitance  per  unit  area. 

The  OTAi  is  operated  a^  a  comparator  and  deddes  the  sign  of  the  state  voltage  I’x  =  «xl-«x2- 
The  system  is  switched  into  the  next  output  state  by  the  dual  nonoverlapping  dock  signals  (p\ 
and  Data  are  read  out  via  the  bus  OUT  when  activating  the  transmissiongate  T3.  The  initial 
values  are  loaded  by  the  bus  IN  using  the  transmission  gate  T5.  For  a  detailed  analysis  refer 
to  [5],  [6].  A  4  by  4  cell  chip  has  been  designed  for  experimental  purposes  using  this  drcuit 
architecture. 


3  Measurement  Results 

In  CMOS  processes  fabrication  tolerances  may  occur,  which  are  related  due  to  small  changes 
of  the  oxide  thickness,  different  donor  concentrations  and  a  possible  fringing  of  structures.  This 
may  cause  different  threshold  voltages  for  the  transistors  leading  to  deviations  of  the  saturation 
currents.  Some  of  the  effects  can  be  suppressed  by  a  spedfic  geometrical  design  in  the  layout, 
but  in  general  they  cannot  be  exduded.  This  section  gives  static  and  dynamic  measurement 
results  of  the  test  chip  described  in  Section  2.  It  has  been  fabricated  on  a  standard  digital  1.5  fi 
single  poly  double  metal  process  at  ES2. 

If  all  weight  voltages  ±vtf‘  and  the  input  voltage  t;|  are  set  to  zero,  a  constant  cell  offset  has 
been  observed,  which  is  ^ven  exemplarily  in  Table  1  for  a  single  chip. 


cell  1: 

cell  5: 

0.21  V 

cell  9: 

0.11  V 

cell  13: 

-0.04  V 

cell  2: 

0.04  V 

cell  6: 

0.13  V 

cell  10: 

0.04  V 

cell  14: 

ceU  3: 

cell  7: 

0.02  V 

cell  11: 

0.16  V 

cell  15: 

0.11  V  1 

cell  4: 

0.00  V 

cell  8: 

0.18  V 

cell  12: 

0.02  V 

Table  1:  Offset  voltage  of  the  cells. 

Cell  1,  3  and  14  have  no  measurement  bus  for  the  state  voltages  Vxl  and  therefore,  their 

offset  cannot  be  determined  directly.  The  cell  offset  is  caused  by  a  mismatch  of  the  differential 
currents  of  each  circuit  component,  which  is  summed  up  in  the  worst  case.  Compared  with  the 
maximum  linear  range  of  ±2  V  of  the  transconductance  amplifiers  OTA2,  this  offset  seems  to  be 
very  large.  However,  it  is  a  constant  value  amd  does  not  change  during  operation.  Hence,  it  can 
be  compensated  with  the  input  voltage  of  the  OTA2.  This  makes  a  complex  compensation 
circuit  superfluous. 
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Fig.  2  shows  the  transfer  characteristic  of  the  offset  compensated  OTA2  for  13  cells,  if  all  weight 
voltages  are  set  to  zero.  The  bias  voltage  of  the  OTA2  has  been  chosen  to  vi,2  =  —1.35  V  and 
vta  was  set  to  -2.0  V.  The  power  supply  amounts  to  =  2.5  V  and  =  -2.5  V. 


Figure  2;  Dc  transfer  characteristic  of  the  OTA2  for  13  cells. 


in  V 


v,“* in  V 


Figure  3:  Dc  transfer  characteristic  of  a  single  conductance  multiplier  for  13  cells. 

Since  the  saturation  voltages  are  slightly  different,  their  slopes  have  identical  values  witliin  a 
linear  range,  which  stretches  between  -0.5  V  and  0.5  V.  The  slope  of  the  OTA2  can  be  adjusted 
by  the  bias  voltage  V{,2-  Then,  the  saturation  voltage  is  shifted  between  0  V  and  2  V,  too,  while 
the  linear  range  remains  unchanged.  It  corresponds  to  a  gain  between  0  and  3.0  of  the  input 
voltage.  Because  the  resistors  Rx  have  been  designed  to  a  resistance  of  200  kQ,  a  maximum 
transconductance  gm  =  15  fiS  has  been  obtained  for  the  OTA2. 
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In  Fig.  3  the  dc  transfer  characteristic  of  a  single  weight  coefficient  is  illustrated  for  13  cells.  The 
remaining  feedback  coefficients  are  set  to  zero  and  the  input  voltage  compensates  the  offset. 

The  linear  range  stretches  between  —0.9  V  and  0.9  V  with  a  gain  of  0.8.  This  corresponds  to 
a  maximum  absolute  value  of  5  (iS  for  when  inserting  the  process  and  design  parameters 
in  (5).  No  significant  deviations  have  been  observed  for  different  weight  coefficients.  This  is  also 
true  for  a  comparison  of  the  characteristics  between  different  chips. 

The  network  variables  and  parameters  are  retransformed  by 


Rxy 


x%k)  = 


v.%kT) 

Vsat  ’ 


y%k)  = 


v^jkT) 

V,at  ’ 


LC 


(6) 


to  obtain  the  algorithm  (1),  (2).  The  constraint  y'^{k)  =  ±1  implies  V^at  =|  Vy{kT)  |=  v'y  —  Vr  = 
0.8F.  This  leads  to  k‘^  6  [—2.5, 2.5]  and  x‘^(k)  €  [—3.75,3.75]  if  a  linear  range  of  ±1.5V'  is  assumed 
for  v^i  and  v^2-  For  the  feedback  coefficients,  €  [—1,1]  is  obtained  from  (5)  and  (6).  Notice 
that  the  output  voltage  t/y  can  be  used  for  scaling,  too.  It  adjusts  the  range  for  k‘^  and  x^^, 
because  it  determines  the  reference  voltage  Vaat-  Then,  the  power  supply  of  the  inverters  I3  and 
I4  in  Fig.  1  consists  of  a  scaling  voltage  Vacal- 

As  an  example  for  dynamic  testing,  the  connected  component  detector  was  chosen  with  the 
initial  pattern  on  the  left  side  of  Fig.  5.  A  cell  bias  was  loaded  to  compensate  the  constant  cell 
offset  and  the  weight  coefficients  have  been  set  to  the  values  in  Fig.  4. 


Figure  4:  Weight  voltage  of  the  feedback  coefficients  in  V. 

Fig.  5  shows  the  time-dependent  development  of  the  output  pattern.  After  four  iterations  all 
outputs  are  constant.  The  drcuit  could  be  operated  even  with  a  clock  frequency  of  10  MHz. 
However,  for  smaller  weight  coefficients  the  maximum  clock  rate  goes  down. 


m  ,0)  K2)  r(3)  >(4) 

Figure  5:  Time-dependent  development  of  the  output  pattern. 


4  Conclusion 

The  mesurements  have  shown  that  the  applied  architecture  was  well  suited  for  the  realization 
of  discrete-time  cellular  neural  networks.  Especially  the  large  computation  speed  demonstrates 
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the  performance  of  this  network  structure.  Improvements  are  desirable  in  the  linear  range  of  the 
transconductance  amplifier,  because  this  device  needs  a  large  operation  range  to  compensate  the 
output  currents  of  all  conductance  multipliers  as  required  for  many  applications.  Disturbances 
in  the  input  voltage  imply  a  large  deviation  of  the  state  voltage,  since  they  are  amplified  by 
the  gain  of  the  OTA,  which  may  be  much  larger  than  the  gain  of  the  multipliers.  Hence,  the 
bias  voltage  of  the  OTA  should  be  adjusted  so  that  the  maximum  requested  absolute  value  of 
the  cell  bias  is  yet  reached.  Further  improvement  of  the  circuit  properties  could  be  achieved  by 
using  cascoded  current  mirrors.  Since  their  transistor  geometries  can  be  chosen  very  small  the 
size  of  a  cell  is  not  significantly  increased. 
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Abstract 

A  general  technique  for  continuous  time  (CT)  and  discrete  time  (DT)  Cellular  Neural  Networks 
(CNNs)  implementation  using  current-mode  techniques  is  presented.  The  proposed  methodol¬ 
ogy  yields  high  pixel  densities,  high  speed  and  low  power  consumption,  while  the  design  proce¬ 
dure  is  extremely  simple.  Resulting  circuits  are  well  suited  for  standard  digital  CMOS 
processes,  since  only  MOS  transistors  are  required.  CNN  cell  layouts  are  shown  together  with 
electrical  simulation  results  from  extracted  netlists  of  complete  networks.  Montecarlo  analysis 
demonstrates  the  viability  of  the  proposed  techniques.  Basic  building  blocks  have  been  experi¬ 
mentally  tested. 

Introduction 

Cellular  Neural  Networks  are  two-dimensional  arrays  of  locally  interconnected  cells.  Each  cell 
in  a  CNN  has  three  associated  variables,  namely: 

•  Cell  state:  xf(t),  which  conveys  cell  energy  information  as  a  function  of  time. 

•  Cell  output:  y^(t),  which  is  obtained  from  the  cell  state  via  a  nonlinear  limitation: 

/(O  =  5(k(0+l|-k"(0-l|)  =/U"(0)  (1) 

•  Cell  input:  u^,  representing  external  excitation. 

The  behavior  of  the  network  is  governed  by  a  system  of  dynamic  equations,  one  for  each  cell, 
which  in  the  original  model  are  given  by  [ChuaSSa]: 

+Z)^'+  X  +5^}  Vce  Cj<D  (2) 

where  the  coefficients  A/  and  B/,  with  d  eN,.(c),  can  be  arranged  into  two  matrices,  called  the 
feedback  and  control  templates  respectively,  and  is  the  ojfset  term.  The  time  constant  x  is 
invariant  from  cell  to  cell. 

The  CNN  description  summarized  above  corresponds  to  a  CT  system.  The  same  input/output 
mapping  can  be  obtained,  under  some  re.strictions,  from  a  DT  system  obtained  using  some 
explicit  integration  algorithm.  The  simplest  choice  is  Forward  Euler,  which  results  in 

-I- 1)  =  [-/'(«) +D‘^+  5^  Vc€  ^2)  (3) 

</€N,(c)  J 
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where  n  denotes  the  DT  instant  and  T  is  the  time  interval  between  DT  instants.  Stability  of  this 
system  and  accuracy  in  the  emulation  of  the  CT  model  is  strongly  influenced  by  the  ratio  Tlx, 
which  also  influences  the  number  of  DT  instants  required  for  convergency.  A  good  heuristic 
choice,  which  has  always  given  correct  results  in  our  trials  and  which  significantly  simplifies 
cell  design  is  T/x=l.  With  this  assumption,  (3)  can  be  written  as 


/(«  +  !)  =/ 


D‘ 


d  e  (c) 


{A‘yM+By} 


Vc  e  gi) 


(4) 


where /('.j  is  the  nonlinear  function  in  (1).  Note  that  state  variables  do  not  appear  explicitly  in 
(4)  and,  hence,  they  are  not  required.  In  this  case,  signal  ranges  of  all  variables  in  the  circuit  are 
the  same,  as  opposed  to  the  original  CT  model  in  which  a  higher  signal  range  is  needed  for  state 
variables  [Chua88a]. 

Practical  use  of  CNNs  in  any  of  its  potential  application  fields  {Chua88a,b]  requires  feasible  and 
efficient  implementation  techniques.  Major  trends  are  area  and  power  efficiency,  operation 
speed,  testability  and  ease  of  communication  with  the  outer  world.  Current-mode  techniques 
[Toum90]  are  propo.sed  for  CNNs  implementation  in  this  paper.  The  results  obtained  show  bet¬ 
ter  area,  power  and  speed  figures  than  previous  proposals  [Cruz91,  Halo92,  Harr92]. 


Current  mode  basic  building  blocks 

The  operations  required  for  the  implementation  of  (2)  and  (4)  are:  summation,  weighted  repli¬ 
cation,  nonlinear  limitation,  integration  (CT),  and  one  clock-cycle  delay  (DT). 

Summation  is  achieved  in  current-mode  by  simply  routing  different  signals  to  a  common  node. 
Weighted  replication  is  obtained  by  using  a  simple  analog  circuit:  the  current  mirror.  In  MOS 
technologies,  the  weight  is  controlled  by  the  sizes  of  tlie  input  and  output  transistors  in  the  mir¬ 
ror.  Since  in  current  mirrors  currents  always  flow  in  the  same  direction,  bias  shifting  is  required 
to  allow  for  the  symmetric  existence-interval  of  the  variables.  These  concepts  are  illustrated  in 
Fig.  1(a)  and  (b).  Several  output  currents  with  the  same  or  different  scaling  factors  can  be 
obtained  by  using  different  output  transistors. 

In  Fig.  1(b),  a  saturation  nonlinearity  appears  as  a  result  of  the  cut-off  of  the  input  transistor.  By 
cascading  two  of  these  structures,  the  nonlinearity  in  (1)  is  obtained,  as  shown  in  Fig.2. 

The  basic  dynamic  block  for  DT  systems  is  obtained  from  the  circuit  in  Fig.2  by  introducing 
switches  in  the  gate-voltage  paths  of  the  current  mirrors,  as  shown  in  Fig.3.  Each  switch  is 
driven  by  one  of  two  nonoverlaped  clock  signals,  yielding  a  full  cycle  delay. 

Finally,  Fig.4  shows  a  CT  lossy  integrator  which  performs  according  to  the  following  equation 


X 


dl, 

dt 


(5) 


where  x  is  given  by  x=Clg,„,  g,„  being  the  small  signal  transconductance  of  the  transistors  in  the 
current  mirror.  Although  the  time  constant  x  is  a  function  of  the  output  current  (through  g„,),  it 
can  be  shown  that  this  does  not  affect  the  dynamic  properties  of  the  cell.  Bias  current  in  the  inte¬ 
grator  must  be  adjusted  to  provide  enough  range  for  stale  variables.  An  expression  for  this 
range,  usually  between  5  an  10  times  that  of  the  output  variables,  is  given  in  [ChuaSSaj.  In  addi¬ 
tion  to  the  integrator,  the  block  that  loads  it  must  be  capable  of  sinking  (sourcing)  currents  in 
the  same  range. 
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Cell  architecture 

Designing  CNN  cells  described  by  (2)  or  (4)  is  straight  forward  using  tlie  blocks  described  in 
the  previous  section.  Fig. 5  shows  a  schematic  diagram  of  a  cell  valid  for  both  approaches. 
Weighted  replication  is  achieved  with  the  circuit  in  Fig.  1(b).  When  sign  inversion  is  required, 
an  additional  replication  stage  is  ca.scaded.  In  the  CT  case,  the  dynamic  block  consists  of  a  CT 
integrator  loaded  witli  a  bias  sliifted  current  mirror.  Both  the  integrator  and  its  load  must  have 
enough  signal  range  for  the  state  variable.  The  nonlinear  limiter  truncates  the  state  variable  to 
the  limits  of  the  output  variables.  In  the  DT  case,  the  dynamic  block  consists  of  the  delayer  in 
Fig.3.  This  block  also  acts  as  a  nonlinear  limiter.  Hence,  the  limiter  block  in  Fig. 5  can  be  elim¬ 
inated. 

For  simplicity,  I/O  circuitry  has  not  been  included  in  the  diagram  in  Fig.5.  Initial  state  values 
x‘^'(0)  and  external  inputs  if  can  be  electrically  set,  or  optically  transmitted  using  photoactive 
devices  [Sayl91  ].  Output  of  each  ceil  can  be  evaluated  with  the  help  of  an  additional  replication 
branch.  Funh.er  I/O  considerations  are  left  a.side  in  this  paper  due  to  the  limited  available  space. 
Note  that  weighted  replication  is  performed  at  the  output  of  each  cell.  In  other  words,  each  cell 
produces  a  different  output,  with  the  required  weight,  for  each  neighbor.  At  each  cell,  neighbor 
contributions  are  summed  at  the  input  node  as  they  are  received.  In  order  to  avoid  confusion 
when  designing  current-mode  CNNs,  it  is  convenient  to  work  with  new  template  matrices, 
obtained  from  tlte  original  ones  by  interchanging  entries  along  all  radial  lines  of  the  matrices. 
To  conclude  this  section.  Fig. 6  shows  two  complete  examples  of  cell  schematics,  one  for  CT 
and  another  for  DT.  Both  are  designed  for  connected  component  detection  (CCD).  Note  the 
reduced  complexity  achieved  with  the  DT  approach,  due  to  the  common  ra.^'ge  of  state  and  out¬ 
put  variables.  This  results  in  significantly  better  area  an  power  figures.  On  the  other  band,  prob¬ 
lems  associated  to  DT  analog  circuits  must  be  taken  in  account  (feedthrough,  noise).  A  new  CT 
CNN  model  which  combines  the  advantages  of  the  DT  approach  is  described  in  a  separate  paper 
[Huer92]. 

Results 

Several  prototype.s  have  been  designed  following  the  proposed  technique  on  a  l.b.um  single¬ 
poly  double-metal  digital  CMOS  proces.s.  As  an  example,  results  of  a  CT  CNN  for  CCD  will  be 
described  here.  The  design  actually  includes  two  prototypes,  each  with  sixteen  cells  in  a  row. 
First  prototype  (PI)  corresponds  to  tlie  .schematic  in  Fig. 6(a)  while  the  second  one  (P2)  corre¬ 
sponds  to  the  simplified  CT  model  mentioned  above,  whose  schematic  can  be  viewed  as  that  of 
Fig. 6(b)  when  both  switches  arc  simultaneously  o/i.  Area  and  power  figures  of  prototype  P2  are 
essentially  the  same  thaii  for  a  DT  design. 

Design  process  begins  by  choosing  a  unitary  bias  enrrent  (Iq)  which  in  our  case  was  2|i.A.  A 
current  mirror  capable  of  driving  21  q  and  a  fg  current  .source  must  then  be  designed.  In  our  pro¬ 
totype,  cascode  structures  were  u.scd  to  implement  both  the  current  mirror  and  the  current 
source,  all  transistors  ha'dng  vi’//=4.{)pm/3.2fim.  After  this  steps,  implementing  any  CNN  is  just 
a  matter  of  combining  the  same  building  blocks. 

Power  calculation  is  extremely  simple  from  cell  schematic.  With  a  5V  power  supply,  PI  con- 
.sumes  290pW/cell,  while  P2  takes  only  9()|iW/cel!.  Fig. 7(a)  and  (b)  show  the  layouts  of  PI  and 
P2  single  cells  respectively,  including  the  initialization  circuitry,  an  extra  output  for  evaluation 
purpo.se,  and  the  necessary  spacing  and  lines  to  be  connected  by  abutment  with  neighboring 


171 


cells.  Area  figures  are  16.368|im^/cell  (equivalently  61  cells/min^)  for  PI,  and  8303|im^/cell 
(equivalently  120  cells/mm^)  for  P2. 

Fig.8  shows  simulated  electrical  results  from  the  layout-extracted  netlist  of  ihe  prototype  P2, 
which  is  slightly  faster  than  PI.  In  order  to  test  the  prototypes  with  digital  testing  equipment, 
interfaces  are  u.sed  in  the  design  so  that  chip  I/O  is  perfonned  in  voltage  form.  In  Fig.8,  high 
voltages  (5  V)  correspond  to  +2)iA,  while  low  voltages  (OV)  correspond  to  -2|iA.  Boundary  cells 
are  set  to  a  low  state  value.  Note  that  convergency  time  is  only  3.5ps,  with  16  cells  in  a  row. 
Finally,  Montecailo  simulations  showed  90%  and  100%  insensibility  to  process  and  geometries 
variations  (out  of  30  trials)  for  prototypes  PI  and  P2  respectively. 

Conclusions 

A  general  procedure  for  the  design  of  continuous  and  discrete  time  CNNs  using  current-mode 
techniques  has  been  presented.  The  circuits  obtained  with  this  methodology  are  advantageous 
in  terms  of  area,  speed  and  power  over  previous  implementation  techniques.  Several  prototypes 
have  been  designed,  the  results  been  highly  insensitive  to  process  and  design-parameter  varia¬ 
tions. 


References 

[Chua88a]  L.O.  Chua  and  L.  Yang;  “Cellular  Neural  Networks:  Theory”.  IEEE  Trans.  Circuits 
and  Systems,  Vol.  CAS-35,  pp  1257-1272,  1988. 

[Chua88b]  L.O.  Chua  and  L.  Yang:  “Cellular  Neural  Networks;  Applications”.  IEEE  Trans. 
Circuits  and  Systems,  Vol.  CAS-35,  pp  1273-1290, 1988. 

[Cruz91]  J.M.  Cruz  and  L.O.  Chua:  “A  CNN  Chip  for  Connected  Component  Detection”. 
IEEE  Trans.  Circuits  and  Systems,  Vol.  CAS-38,  pp  812-817, 1991. 

[Halo92]  K.  Halonen  et  al:  “VLSI  Implementation  of  a  Reconfigurable  Cellular  Neural  Net¬ 
work  Containing  Local  Logic”.  Int.J.  CircuitTheory  Applications,  1992  (to  appear) 

[Harr92]  H.  Han'eretal.;  “An  Analog  Implementation  of  Discrete-Time  Cellular  Neural  Net¬ 
works”.  IEEE  Trans.  Neural  Networks,  Vol.  3,  pp  466-476,  1992. 

[Huer92]  J.L.  Huertas  et  al.:  “Analog  VLSI  Implementation  of  Cellular  Neural  Networks”. 
Second  Int.  Workshop  on  CNNs  and  their  Applications,  Munich,  1992. 

[Sayl91]  A.H.  Sayles  et  al.:  “An  Optoelectronic  CMOS  Memory  Circuit  for  Parallel  Detec¬ 
tion  and  Storage  of  Optical  Data”.  IEEE  J.  Solid  State  Circuits,  Vol.  26,  pp  1110- 
1115,  1991. 

[Toum90]  C.  Toumazou  et  al.:  "Analogue  IC  Design:  the  Current  Mode  Approach" .  Peter  Per¬ 
egrines  Ltd.,  London  1990. 


Figure  I :  a)  MOS  current  mirror:  h)  MOS  current  mirror  with  bias  shifting 
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Figure  7:  Cell  layout  for  two  CCD  prototypes:  a)  prototype  PI ;  b)  prototype  P2 
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Figure  8:  Electrical  simulation  results  from  extracted  netlist  of  prototype  PI 
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Abstract 

This  paper  is  concerned  with  an  optical  implementation  of  Cellular  Neural  Net¬ 
works,  defined  by  a  template  with  non-zero  feedback  operator.  Two  design  rules 
which  consider  a  transient  duration  of  optically  realized  CNNs  are  formulated.  A 
possibility  of  an  increase  in  a  resolution  of  images,  which  are  to  be  processed  in  the 
optical  system,  is  also  discussed.  The  rules  formulated  in  the  paper  are  applied  into 
a  CNN  design  procedure  -  an  optical  iir.plementations  of  two  distinct  Hole  Pilling 
CNNs  and  a  Shadow  Detecting  CNN  arc  analyzed. 


1  Introduction 

The  Cellulax  Neural  Network  paradigm  (abbreviated  henceforth  CNN)  [1]  has  attracted 
a  lot  of  interest  because  it  offers  a  natural  way  of  solving  many  of  image  processing 
problems  and  a  feasilxUty  of  simple  CNN  circuits  (as  e.g.  [2])  VLSI  implementation. 
However,  problems  with  implementing  highly  interconnected  CNNs  composed  of  large 
number  of  cells  are  severe.  An  alternative  mean  for  a  physical  CNN  realization,  which 
allows  for  overcoming  these  difficulties,  is  the  optical  system  proposed  in  [3]  and  [4]. 
Specific  properties  of  the  optical  system  should  he  considered  in  deriving  CNN  applications 
which  arc  to  be  realized  optically,  since  then  we  can  exploit  the  system  advantages  and 
minimize  its  drawbacks. 

The  main  subject  of  this  paper  is  a  determination  of  ways  which  allow  for  increasing  a 
processing  speed  of  optically  implemented  feedback  CNNs.  A  CNN  is  referred  to  as  a 
feedback  one  if  there  i.s  at  least  a  single  non-zero  element  in  the  feedback  operator  (A)  of 
the  circuit  template  (see  [1]).  Another  issue  which  is  discussed  in  the  paper  is  a  possible 
increase  in  a  resolution  of  images  which  arc  to  be  processed. 

I'he  main  results  of  the  paper  are  following: 

•  Design  rules  which  can  be  incorporated  into  a  CNN  design  procedure  and  which 
concern  a  convergence  speed  of  optically  realized  feedback  CNNs  are  presented. 
Conditions  which  allow  for  an  increase  in  a  resolution  of  images  which  are  to  be 
processed  in  the  system  are  formulated. 
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The  paper  has  the  foUowiiig  structure.  Basic  processiug  properties  of  the  optical  system 
are  outlined  in  Section  2.  The  design  rules,  which  concern  a  processing  speed  of  opti¬ 
cally  realized  CNNs,  are  formulated  in  Section  3.  Section  4  considers  a  possibility  of  an 
increase  in  a  resolution  of  images  which  are  to  be  processed  in  the  system.  The  results 
of  presented  considerations  are  applied  to  analyze  an  optical  implementation  of  some  of 
existing  feedback  CNN  applications  in  Section  5. 


2  Processing  properties  of  the  optical  system 

The  optical  system  implements  physically  a  CNN  model  which  combines  the  discrete- 
time  version  of  the  cell  state  equation  [l]  (where  a  time  step  is  chosen  to  be;  At  =  ^,  as 
proposed  in  [5]): 


Vxij(n  -H  1)  =  +13  13  ‘  ^  (1) 

k=—T  l=—r  k=—T  l=—T 

and  a  sigmoidal  output  transforming  function,  which  approximates  the  cell  output  equa¬ 
tion  [1].  The  symbols  Vxi^y  Vpijy  Vuijy  AutBiUyl  have  the  meaning  defined  in  [1]. 

The  most  attractive  properties  of  the  optical  system  are:  a  realizability  of  CNNs  composed 
of  a  large  number  of  cells  (which  means  a  possibility  of  high  resolution  image  processing) 
and  an  ease  of  large  neighborhood-size  templates  realization.  However,  it  should  be  no¬ 
ticed,  that  due  to  a  special  coding  scheme  ([3], [4]),  a  maximum  resolution  of  images  which 
can  be  processed  in  the  system  is  generally  four  times  lower  than  the  availaUe  reseduiion 
of  the  system  input  device  (Liquid  Crystal  Light  Valves  array). 

A  presence  of  feedback  allows  for  explmting  a  global  information  by  every  circuit  cell, 
despite  a  local  interconnection  pattern.  In  the  case  of  discrete-time  CNNs  this  interac¬ 
tion  at  large  distances  occurs  indirectly,  through  cell  outputs,  in  consecutive  iteratiems. 
Realization  of  every  iteration  requires  closing  a  feedback  loop,  i.e.  transferring  results 
obtained  in  some  earlier  step  from  the  system  output  back  to  its  input.  This  is  highly 
time-consuming  operation,  which  severely  slows  down  a  system  processing  speed  (compu¬ 
tations  are  performed  at  the  speed  of  light),  and  can  be  regarded  as  the  main  drawback  of 
the  optical  system.  Therefore  the  following  conclusion  concerning  processing  properties 
of  the  optical  system  can  be  formulated; 

•  Applications  which  require  a  large  neighborhood-size  and  a  small  number  itera¬ 
tions  to  complete  a  network  transient  are  well  suited  for  the  optical  CNN  realization. 

3  A  processing  speed  of  optically  realized  feedback 
CNNs 

The  optical  system,  despite  its  drawbacks,  can  ^pear  the  only  physical  mean  of  some 
CNN  circuits  realization.  Therefore  it  is  necessary  to  investigate  possible  ways  oS  improv¬ 
ing  system  properties,  in  particular,  ways  of  increasing  its  processing  speed.  Some  CNN 
design  methods  for  feedforward-only  CNNs  has  been  presented  in  [6].  This  paper  focuses 
on  a  feedback  CNN  template  design. 

Let  us  consider  the  following  process; 
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a  cell  Ctj  output  follows  a  transition  from  ’-1’  to  *+l’,  which  occurs  in  the  output  of  any 
of  its  neighboring  cells^  unless  some  condition  related  to  inputs  analyzed  by  a  cell  Cij  is 
satisfied,  - 

aaid  let  us  refer  it  to  as  a  high  output  propagation.  A  low  output  propagation  is  understood 
analogously.  These  phenomena  are  present  in  many  CNN  applications  introduced  to  date 
([7],  [8],  [9]).  If  a  CNN  which  is  to  be  implemented  optically  involves  a  propagation 
phenomenon,  it  should  be  designed  in  a  way  which  ensures  completing  this  process  in  a 
minimum  possible  number  of  iterations.  Therefore,  some  means  which  allow  for  obtaining 
this  goal  should  be  included  into  a  CNN  design  procedure. 

A  simple  analysis  of  the  propagation  phenomena  allows  to  determine  some  template  prop¬ 
erties  which  allow  for  an  increase  in  a  processing  speed  of  optically  realized  CNNs.  First, 
observe  that  if  a  given  cell  is  allowed  to  change  its  output,  this  change  should  have  the 
majfimum  possible  magnitude  in  response  for  any  change  occurring  in  outputs  of  its  neigh¬ 
borhood.  This  means  that  a  change  of  an  absolute  value:  AVy  =  Vymas  —  Vy^in  (i-c. 
AVy  =  2  (oi  Vj/min  =  —  1  and  Vy^aa  =  +1)  should  occur  in  a  single  iteration.  This  leads 
to  the  following  conclusion,  which  can  be  considered  as  an  auxiliary  CNN  design  rule: 

Rule  1  Suppose,  that  we  have  a  processing  problem  which  involves  a  propagation  of  high 
or  low  output  and  which  is  to  be  solved  in  an  optically  realized  CNN. 

Template  element  values  should  be  chosen  to  ensure  a  binary  output  from  any  cell  in 
response  to  any  combination  of  cell  controlling  signal  values  which  can  appear  in  this 
network,  i.e.: 

I'^Au-V yk+i,t+)(n)  +  51  •  V Uk+ij+j  +  I\>1  (2) 

M  kl 

Note,  that  if  this  condition  is  satisfied,  all  cell  outpnts  are  binary  during  a  processing,  so  a  CNN 
behaves  as  if  it  is  composed  of  elements  with  ta  step-like  output  nonlinearity 
Observe  further,  that  if  we  increase  a  cell  neighborhood-size  then  a  cell  can  respond  for  a 
change  occurring  in  more  distant  location.  This  reasoning  leads  to  a  conclusion: 

Rule  2  Maximum  neighborhood  size,  which  allows  for  realizing  given  processing  prob¬ 
lem,  should  be  considered  in  deriving  CNN  templates  to  ensure  minimizing  of  a  transient 
duration  in  optically  realized  feedback  CNNs. 

4  A  resolution  problems  in  optically  realized  feed¬ 
back  CNNs 

Let  us  consider  a  coding  scheme  for  signals  which  axe  to  be  processed  in  the  optical  system, 
presented  in  [3], [4],  An  input  information  loaded  into  the  system  is  stored  according  to  this 
scheme  in  an  input  Liquid  Crystal  Light  Valve  array  (abbreviated  LCLV).  Half  of  all  of  the 
available  LCLV  array  elements  arc  used  for  storing  valuen  which  are  to  provide  a  required 
shift  in  a  signal  dj'namic  range.  This  shift  includes  a  CNN  bias  (/),  but  primarily  it  is  to 
allow  for  a  discrimination  between  negative  and  positive  results  of  optical  computations. 
A  result  of  the  optically  computed  cell  state  (l)  is  represented  by  a  light  vector 
which  has  the  magnitude  pven  by: 

Vx‘,j(n  +  1)  k  ■  (Yi  >4«(Vj/x:+.,/+y(n)  -b  1)  -h  X!  BkiiVuk+r,,+j  -f  1)  -f  d  -b  7)  (3) 

ki  k! 
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where  k  is  some  positive  constant,  d  represents  an  additional  shift,  and  terms  ’+1’  are 
introduced  because  transmissivities  of  light  valves,  which  are  propKirtional  to  CNN  signals, 
are  non-negative. 

Let  us  assume  that  a  CNN  bias  (/),  which  can  be  considered  as  a  shift  in  a  dynamic 
range  of  the  state  equation  (1),  is  realized  in  an  electronic  feedback  path  of  the  system 
(by  means  of  a  level  shifter)  rather  then  in  an  optical  signal  path.  It  can  be  shown  that 
errors,  caused  by  inability  of  discrimination  between  negative  and  positive  magnitudes  of 
vectors  Vx[j^  can  be  neglected  in  a  signal  processing  if  the  following  relation  holds: 

>  \v^LJ  (4) 

where  Vx^_i  is  a  magnitude  of  a  light  vector  which  corresponds  to  the  value  - 1  ^  of  the  state 
equation  (1);  is  a  magnitude  of  the  light  vector  which  corresponds  to  a  Tninirnnm 

value  of  the  state  equation. 

Note  that,  if  there  is  no  entries  which  code  a  bias  and  shift  in  the  LCLV  array,  a  zero-level 
of  optically  computed  cell’s  state  is  in  fact  shifted  by  a  value  of:  k  •  (2  -Au  +  52  Bfu  —  I). 
Thus,  magnitudes  of  VxLi  and  Vx[^^^  can  be  evaluated  using  the  formulas: 

VxL,=:k-iEAu  +  EBu-I-l) 

=  *  •  (-2 E  \Au\  -2^\Bu\- I)  ioiAu,Bu<0 

The  following  conclusion  can  be  formulated: 

•  If  the  light  vectors  given  in  equation  (5)  satisfy  the  relation  (4)  then  it  is  possible 
to  realize  a  bias  in  an  electronic  part  of  the  system,  and  consequently,  to  double  a 
re.solution  of  images  which  are  to  be  processed  in  the  system. 

Observe,  that  if  the  relation  52  Au  +  E  —  I  hcJds,  and  the  condition  (4)  is  satisfied, 
then  no  shift  is  required  in  the  system. 

5  A  design  of  optically  realized  CNNs 

Templates  for  two  of  already  existing  CNN  applications  are  analyzed  in  this  section  in 
the  way  which  takes  into  account  a  convenience  of  their  optical  implementation.  The 
CNNs  considered  are:  the  Hole  Filler,  presented  in  [7]  and  [10],  and  the  Shadow  Detector, 
presented  in  [7]. 

5.1  A  modification  of  the  Hole  Filler  template 

An  analysis  of  the  Hole  Filler  operation  shows,  that  only  the  first  of  the  formulated  rules 
can  be  apphed  into  a  template  design  procedure.  One  can  notice,  that  comparing  two 
proposed  hole  filler  templates  ( [7]  and  [10]),  only  the  first  one  (proposed  by  Matsumoto 
et  all.)  complies  to  the  first  rule,  so  it  is  ex{}ected  to  ensure  the  faster  convergence  in 
the  optical  system.  Results  of  computer  simulations  which  verify  this  hypothesis  are 
shown  in  Figure  1.  An  image  which  is  to  be  processed  is  given  in  Figure  la,  a  processing 
result  is  shown  in  Figure  Ic.  This  is  obtained  in  50  iterations  when  the  first  of  considered 
templates  is  used,  while  it  takes  60  iterations  to  reach  this  result  in  the  latter  case.  Figure 
lb  presents  the  CNN  output  after  50  iterations  in  the  case  when  the  template  given  flO] 
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Figure  1:  A  comparision  of  the  hole  filling  operation 

is  used.  Initial  states  of  all  boundary  network  cells  are  set  to  ’-1’  and  of  the  remaining 
ones  are  set  to  *+F. 

Observe,  that  template  elements  of  both  hole  filler  templates  satisfy  the  condition  (4): 

Kii  =  10  in  the  former  case,  and  =  0,  Vi^  =  10.5  in  the  latter  case. 
Therefore,  if  we  realize  an  appropriate  signal  range  shift  in  an  electronic  path  of  the 
system,  a  resolution  of  images  which  are  to  be  processed  can  be  doubled. 

5.2  A  modification  of  the  Shadow  Detector  template 

The  second  example  shows  the  application  of  both  rules  in  order  to  derive  an  alternative 
template  of  a  shadow  detector.  Let  us  assume  that  we  increase  a  neighborhood  size  of 
a  template  (i.e.  r  —  2  instead  of  r  =  1  as  it  was  given  in  [7]).  Applying  the  first  of  the 
formulated  rules  into  a  design  strategy  proposed  in  [10],  the  following  element  values  can 
be  obtained: 

A=[0  0  J  1  l]  1  =  2 

A  processing  example  given  in  Figure  2  compares  an  operation  of  a  CNN  defined  by 
this  template  and  an  operation  of  the  original  Shadow  Detecting  CNN.  An  image  to 
be  processed  is  presented  in  Figure  2a,  the  processing  result,  which  is  obtained  in  the 
former  case  after  14  and  in  the  latter  case  after  28  iterations,  is  given  in  Figure  2c,  while 
the  output  of  the  original  Shadow  Detector  after  14  iterations  is  shown  in  Figure  2b. 
Appropriate  templates  for  neighborhood  size  larger  than  r  —  2  can  be  easily  derived. 
A  correct  processing  can  be  obtained  if  it  is  assumed  that  after  every  iteration  right 
boundary  ceU  states  (outputs)  are  set  to  '-V  regardless  of  computation  results. 

Note,  that  a  resolution  of  images  to  be  processed  in  the  case  of  proposed  template  equals 
a  resolution  of  a  LCLV  device  if  an  tqipropriate  shift  is  realized  in  the  feedback  path  of 
the  system. 

6  Conclusions 

Possible  methods  of  improving  processing  properties  of  optically  realized  CNNs  were  pre¬ 
sented  in  the  paper.  An  application  of  two  design  rules,  which  has  been  formulated,  can 
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Figure  2:  An  operation  of  a  shadow  detection 

result  in  an  increase  in  the  processing  speed  of  some  optically  realized  feedback  CNN 
applications.  A  realization  of  a  signal  dynamic  range  shift  in  an  electronic  feedback  path 
of  the  system  can  allow  for  an  increase  in  a  resdution  of  images  which  are  subject  to 
optical  processing. 
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Abstract 

The  programmability  (as  a  stored  program)  of  the  CNN  Universal 
Machine  is  discussed  first  It  is  shown  why  and  in  which  sense  this 
machine  is  universal.  A  new  type  of  algorithm,  the  analogic  one,  is 
introduced.  The  application  potential  is  reviewed  and  the  biological 
relevance  is  anal^ed.  It  turns  out  that  the  architecture  is  optimal  not 
only  for  silicon  implementations,  however,  many  biological  information 
processing  organs  have  the  same  structure. 


1  INTRODUCTION 

Since  the  invention  of  the  Cellular  Neural  Network  [1]  several  extensions  [2-7, 
etc.]  formed  the  CNN  (cellular  nonlinear  network)  paradigm.  CNN  is  now  an 
established  field.  In  our  companion  paper  in  this  volume  (Part  1)  we  gave  a  taxonomy 
and  introduced  the  architecture  of  the  CNN  Universal  Machine. 

In  this  paper  we  show  the  key  part  responsible  for  implementing  the  analog 
stored  program  concept  (Section  2).  In  Section  3  we  discuss  the  universality  of  this 
machine  and  the  characteristics  of  the  analogic  (dual)  CNN  algorithms.  In  Section  4  the 
key  application  areas  are  reviewed.  Section  5  contains  some  examples  and  motivations 
concerning  the  biological  relevance. 


2  THE  ANALOG  STORED  PROGRAM  IN  THE  CNN  UNIVERSAL  MACHINE 

The  key  issue  in  inventing  the  digital  computer  was  to  represent  the  program 
as  data  (J.von  Neumann).  .A  condition  for  efficient  impiementation  is  that  the  time  for 
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changing  the  instructions  is  normally  smaller  or  comparable  to  the  instruction  execution 
time. 


Considering  the  standard  fully  connected  analog  neural  networks,  it  is  obvious 
that  the  latter  condition  is  unimplementable.  Partly  because  the  storage  space  of  a 
single  program  "instruction"  is  prohibitively  large,  partly  because  the  reprogrammability 
takes  more  time  than  the  execution  time  itself.  This  is  a  point  where  the  local 
connectivity  comes  into  the  picture.  Namely,  if  we  consider  a  CNN  cloning  template  as 
an  analog  instruction  [2d],  then  we  have  only  one  or  two  dozen  analog  values  for  an 
instruction  to  be  stored  even  if  we  have  thousands  of  processors.  On  the  contrary,  in 
case  of  a  fully  connected  analog  synchronous  neural  network  (e.g.  the  Intel  80170  [9]) 
we  have  about  10  000  analog  values  for  64  processors  and  their  reprogrammability 
takes  considerable  time. 


to  all  cells 
controlling  the 
cloning  templates 


to  all  cells 
controlling  LLU 


Figure  1  The  GAPU 
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The  analog  instructions  in  the  CNN  Universal  Machine  are  stored  in  the  analog 
program  register  (APR)  of  the  global  analogic  program  unit  (GAPU).  The  structure  of 
the  GAPU  is  shown  in  Figure  1 . 

The  analog  memory  elements  in  a  single  line  of  the  APR  is  equal  to  the  nonzero 
cloning  template  parameters.  The  selection  of  the  appropriate  template  is  made  by  the 
global  analogic  control  unit  (GACU)  which  is  a  logic  unit.  The  logic  program  register 
contains  the  local  logic  instructions.  The  switch  configurations  of  the  cells,  including  the 
selection  of  the  appropriate  local  analog  output  unit  (LAOU,  defined  in  Part  1)  can  be 
coded  and  stored  in  the  SCR  register  (switch  configuration  register).  The  machine  code 
of  the  analogic  CNN  program  is  stored  in  the  GACU. 

The  high  level  description  of  the  analogic  algorithm  is  translated  by  a  compiler 
into  this  machine  code. 


3  UNIVERSALITY  AND  THE  ANALOGIC  ALGORITHMS 

We  have  shown  [2c]  that  the  game  of  life  can  be  implemented  by  a  CNN.  Since 
the  game  of  life  algorithm  is  equivalent  to  a  Turing  machine,  therefore  the  CNN  is 
universal  in  a  logic  sense.  As  an  analog  operator,  we  have  shown  that  any  nonlinear 
operator  with  fading  memory  can  be  realized  by  delay-type  neural  networks  [8a].  Here, 
the  open  question  is:  how  many  elements  do  we  need? 

The  flexibility  of  the  CNN  is  not  only  theoretical.  Indeed,  using  our  CNN 
Workstation  it  is  easy  to  write  and  run  analogic  CNN  programs.  As  an  example,  Figure 
2  shows  such  an  algorithm  in  a  block  diagram  form.  The  details  of  the  algorithm  can  be 
found  in  [2c].  It  has  been  used  for  detecting  texture  errors  in  textiles  monitored  through 
a  camera.  Here  only  the  structure  of  the  algorithm  is  important.  In  this  algorithm  we 
have  parallel  branches  as  well.  A  key  point  in  the  implementation  of  the  analogic  (dual) 
CNN  algorithms  that  all  intermediate  results  which  will  be  used  later  are  stored  locally 
as  analog  values.  Therefore,  in  the  cell  of  the  CNN  Universal  Chip  a  local  analog  output 
memory  is  used  to  store  as  many  analog  values  as  the  maximum  number  of  parallel 
branches.  The  local  storage  provides  extremely  high  speed.  We  need  to  output  the 
results  after  the  whole  analogic  algorithm  is  completed,  thus  providing  a  high  output 
throughput. 

Now,  we  have  analogic  algorithms,  instructions,  subroutines,  programs, 
compilers.  This  provides  to  build  up  a  CNN  software  library.  The  CNN  universal 
Machine  is  the  workhorse  for  running  the  analogic  CNN  programs. 
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GRAY  SCALE  PICTURE 


FIEBER  ENDS  KNOTS 


Figure  2  The  flow  diagram  of  an  analogic  (dual)  CNN  algorithm 
4  KEY  APPLICATION  AREAS 

In  the  CNN  literature  (see  [7a-7e]  and  the  references  of  Parti)  we  see  how 
many  application  areas  have  been  discovered.  Without  completeness  let  us  list  some 
major  fields: 

-  Image  processing  including  gray  scale  image  inputs 

-  feature  extraction 

-  motion  detection  and  estimation 

-  path  tracking 

-  collision  avoidance 

-  halftoning  including  motion 

-  object  counting  and  size  estimation 
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-  Analyzing  3D  surfaces 

-  detection  minima  and  maxima 

-  detection  areas  where  the  gradients  exceed  a  given  limit 

-  Solving  partial  differential  equations 

-  Reducing  non-visual  problems  to  geometric  maps 

-  thermographic  images 

-  somatosensory  maps  (tactile  sensing) 

-  antenna  array  images 

-  different  medical  maps  and  images 

-  Modelling  biological  vision  and  other  sensory-motor  organs 

System  integration 

Since  presently  the  analog  VLSI  CNN  chips  are  mainly  small  scale  experiments, 
and  not  considering  the  software  tests  of  key  effects  for  different  applications,  the  few 
emerging  real-life  application  projects  known  to  us  are  using  digital  CNN  hardware 
accelerators. 

Detecting  manufacturing  errors  In  a  printed  circuit  board  production  line, 
multifont  character  recognition,  and  optical  tracking  of  prescribed  curves  in  robot 
motion  were  the  three  projects  we  have  started.  The  details  are  published  elsewhere 
[7a-e].  However,  we  have  found  some  common  problems.  Namely, 

-  the  throughput  problem  of  the  input  interface  (the  ideal  solution  is  the  direct  optical 
input  on  the  chip,  or  the  direct  camera  interface  for  the  digital  CNN  hardware 
accelerators), 

-  the  changing  illumination  circumstances  and  colour  effects  at  the  actual  site  (different 
materials  in  the  printed  circuit  manufacturing,  the  different  qualities  of  papers  and  prints 
in  case  of  character  recognition,  and  the  changing  environment  in  case  of  the  robots), 

-  the  throughput  problem  at  the  output,  especially  in  detection  type  tasks, 

-  the  optimal  implementation  of  the  analogic  (dual)  CNN  algorithms  as  to  the  area 
(memory)  and  time  complexity  is  concerned. 


4  BIOLOGICAL  RELEVANCE 

In  many  living  neural  organizations  the  neurons  are  placed  regularly  in  laminae 
packed  upon  each  other,  and  the  neurons  are  locally  connected  within  a  receptive  field. 
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The  functionality  of  the  organ  (e.g.  retina,  LGN  or  visual  cortical  area)  is  determined  by 
the  neuroanatomical  receptive  field  organization  and  by  the  synapse  types.  The  CNN 
model,  invented  for  artificial  electronic  information  processing,  has  the  same  structural 
and  functional  characteristics.  Until  recently,  both  areas  were  developing 
independently.  It  was  our  fortune  that  in  a  nice  and  challenging  collaboration  [25]  it 
turned  out  that  almost  all  relevant  notions  in  both  fields  can  be  matched.  Thus  it  is  our 
hope,  and  indeed  the  reality,  that  some  known  neural  structures  can  be  translated  into 
CNN  models  which  have  the  one-to-one  silicon  chip  implementations.  Moreover, 
apparently,  the  CNN  model  could  play  a  role  in  discovering  hidden  living  functionalities 
as  being  a  unifying  programmable  model  for  many  biological  phenomena  [26].  The 
receptive  field  organization  is  represented  by  the  cloning  template  of  the  CNN. 

A  few  years  ago  there  were  a  couple  of  electronic  circuit  models  discovered 
and  implemented  on  silicon  chips  to  mimic  specific  neural  phenomena.  A  famous 
example  is  the  so  called  silicon  retina  of  Carver  Mead’s  group  at  Caltech  in  Pasadena 
[13].  Indeed  it  is  a  very  special  case  of  a  CNN  model,  a  resistive  grid  characterized  by  a 
simple  cloning  template. 

Neuromorphic  computing  is  nowadays  a  new  challenge  [27]:  how  to  translate 
one-to-one  a  living  neural  structure  into  a  programmable  analog  computing  chip.  The 
CNN  Universal  Machine  and  Chip  is  providing  a  solution.  We  have  found  also  that  the 
famous  triad  synapse  model  [14]  can  be  represented  by  a  simple  CNN  template, 
playing  important  role  in  motion  related  actions  as  well. 

Below  we  show  some  characteristic  examples  [25b].  The  Herring  grid  illusion 
without  the  central  patches  (taking  into  account  the  amacrine  cell  effects)  and  the  arrow 
head  illusion  with  its  CNN  template. 

Another  important  fascinating  area  is  the  so  called  multi  screen  theater  [18]. 
The  fact  is  that  from  the  same  scene,  preprocessed  by  the  eye,  several  different 
retinotopic  maps  are  generated  [17,19,28].  Later,  these  maps  are  combined  to 
discover  more  complex  events.  This  phenomenon  can  be  directly  modelled  and 
computed  by  the  CNN  universal  chip. 
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(C) 

Figure  3  The  Herring  grid  illusion. 

The  input  we  look  at  (a),  the  result  using  a  resistive  grid  (b),  and  the  result  when 
modelling  the  amacrine  ceil  layer  by  delay-type  templates  (c).  Indeed,  we  do  not  see 
the  patches  in  the  middle  of  the  black  squares. 


187 


Figure  4  The  arrowhead  illusion  and  the  CNN  template 

Although  the  distances  between  the  arrowheads  are  the  same  in  both  cases  we  see 

defferent  distances. 
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ABSTRACT  The  cellular  neural  networks  for  extracting  line  segment  features  are 
proposed.  The  features  Include  a  middle  point,  length  and  angle  of  the  line  segment.  Based 
on  these  features,  appropriate  standard  patterns  are  selected.  The  feature  distribution 
of  the  standard  patterns  are  mapped  onto  that  of  the  handwritten  pattern.  The  feature 
mapping  with  structural  constraints  Is  proposed,  which  can  provide  flexible  mapping  and 
very  fast  convergence.  The  feature  mapping  results  are  estimated  based  on  similarity 
between  the  distorted  pattern  and  the  mapped  standard  ones,  convergence  rate  and 
deviation  from  the  standard  patterns.  Computer  simulation  demonstrates  distortion  free 
feature  extraction  and  flexible  feature  mapping. 

I  LNTRODUCTION 

Japanese  Kanjl  characters  have  their  structural  meaning.  They  are  composed  of  certain 
number  of  writing  strokes.  About  3000  characters  are  included  in  the  first  group  of 
Japanese  Industrial  Standard  (JIS),  which  are  daily  used.  Totally,  about  6000  characters 
are  recommended  by  JIS  to  be  used  in  real  world.  They  have  many  similar  structures. 
Structures  themselves  are  also  very  complicated.  For  this  reason,  handwritten  Japanese 
Kanji  character  recognition  is  Inherently  difficult  subject. 

Neural  network  approaches  to  pattern  recognition  are  classified  into  the  following 
categories.  First,  the  distorted  patterns  are  directly  applied  to  the  neural  network. 
Topological  features  are  extracted  through  the  network.  The  pattern  is  recognized  by 
matching  it  with  standard  patterns  [1].  In  the  second  method,  some  distortion  invariant 
features  are  extracted  by  conventional  methods,  and  these  features  are  applied  to 
multilayer  neural  networks,  trained  by  supervised  learning  algorithms  [2], [3].  Third 
method  is  a  combination  model  of  competitive  learning  and  back-propagation,  which  is 
suited  to  large  scale  character  recognition  [4].  In  any  approaches,  complicated  networks 
and  a  long  computing  time  are  required. 

In  this  paper,  a  new  approach  to  handwritten  Japanese  Kanji  character  recognition  Is 
proposed.  It  consists  of  the  cellular  neural  networks,  for  extracting  features  of  line 
segments,  and  structure  Invariant  feature  mapping. 

H  PATTERN  RECOGNITION  SYSTEM 

Figure  1  shows  a  block  diagram  for  the  proposed  Japanese  Kanji  character  recognition 
system.  A  distorted  pattern  is  applied  to  the  I-layer.  It  is  skeletonized  In  the  S-layer.  In 
the  L-layer,  line  segments  are  extracted.  Vertical  lines,  horizontal  lines  and  Inclined 
lines  with  ±45  degrees  are  extracted  In  the  Li,  La,  La  and  L«  networks,  respectively.  In 
the  A-layer,  the  angle  deviation  is  detected.  In  the  TR-layer,  the  line  segments  are  traced 
starting  from  the  end  points.  The  middle  points  and  the  lengths  of  the  line  segments  are 
extracted.  Each  line  segment  is  characterized  by  the  above  three  kinds  of  features.  The 
extracted  line  features  are  gathered  on  the  F-layer.  The  networks  used  above  are 
developed  using  cellular  neural  networks  [5], [6]. 

Comparing  the  number  of  the  line  segments,  which  are  specified  with  three  kinds  of 
features,  appropriate  candidate  of  the  standard  patterns  are  selected.  The  feature 
distribution  of  the  standard  pattern  is  mapped  onto  that  of  the  distorted  pattern,  while 
maintaining  topological  structure  [6], [7].  The  mapping  result  is  estimated  based  on  three 
kinds  of  measures,  similarity,  convergence  rate  and  deviation  from  the  standard  patterns. 
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m  LINE  SEGMENT  EXTIIACTION 


3.1  Li  and  Lh  Networks 

Since  the  Li  network  can  be  replaced  by 
the  La  network,  by  exchanging  row  and 
column,  the  Li  network  is  only  described  in 
this  section. 

Network  Structure; 

Since  the  network  can  be  regarded  as  a 
matrix,  a  unit,  which  locates  on  the  ith  row 
and  the  jth  column,  is  denoted  by  u(l.J). 
Furthermore,  the  Input  and  output  of  u(i.j) 
are  expressed  by  x(I,j)  and  y(l,J), 
respectively.  Connection  weights,  used  in  the 
Lj  network,  are  defined  as  follows: 
s:  Self-loop  of  u(i,J). 

Uvr'-  Bidirectional  connection  weights  between 
u(i,J)  and  u(i,j-k).  Two  units,  whose 
distance  is  k  units,  are  connected  by  avk. 
It  is  independent  from  the  coordinate 
(i,J),  and  is  determined  only  by  the 
distance  k. 

e  :  Threshold  level.  u(i,  j)  is  activated  if  Its 
input  is  greater  than  or  equal  to  6  . 

The  remaining  connection  weights  are  zero. 
Network  Dynamics; 

The  Li  network  state,  which  is  a  set  of  the 
unit  outputs,  is  initially  set  to  be  the  input 
pattern.  The  network  changes  Its  state  as 
described  in  the  following,  resulting  Into  the 
equilibrium  state.  In  this  sate,  only  vertical 
line  segments  with  the  specified  minimum 
length  can  remain. 

x(i.J)  and  yd.J)  are  denoted  by  Xi,  j(n)  and 
yi.  j(n),  respectively.  In  order  to  descries  the 
state  transition  here. 


Fig.  1  Block  diagram  of  Japanese  Kanjl 
character  recognition  system. 


yi,  j(0)  =/ 1,  u(l.j)  Is  Included  in  the  input  pattern  (1) 

lo,  otherwise 

xi,  j(n)  =  syi.  j(n-l)  +  S  avniyi.  j-k(n-l)  ♦  yi.  j+kin-l)],  n^l  (2) 

H 

If  X,,  j(n)S  0  ,  then  yi.j(n+l)=l  (3a) 

If  Xi.  j(n)<  0  ,  then  yi.  j{n+l)=0  (3b) 


Conditions  for  Extracting  Line  Segments: 

Conditions  for  extracting  lines  segments,  with  the  minimum  length  of  m-units,  are  given 
In  the  following. 


•  Extract  line  segments,  with  the  minimum  length  of  m-units: 


p  m— p— 1 

xi(i.j) 

X2(i.j) 

X3(i.j) 

Xi(l,  j)  =  s  +  S  2avk  ♦  S  avk  ^  0  •  p=0,l,2,... 

k-l  k— p+l 

..[(m-l)/2] 

(4) 

P=0  •  P=0  • 

1  •  1  • 

p=0  • 

0  • 

p=0  • 

I  • 

•  Reject  short  line  segments: 

0  •  2  • 

2  O 

p  m— p— 2 

1  • 

1  • 

X2(i,  j)  =  s  ♦  Z  2avk  X  avk  <  0  ,  p=0,l,2,... 

.,[(m-2)/2) 

(5) 

0  • 

0  • 

k*i  k— p4>i 

•  Reject  non-line  segments; 

Fig.  3  Relations  between  p  and 
unit  locations,  a  Is  chosen  to 

p  m— p— 1 

..l(m-l)/2] 

(6) 

be  3.  •  active,  O  Inactive. 

X3(1|J)  ~  5*  23lvK  *  21  Avk  ^  ^  •  P”0pli2,«* 

k-l  K-P+l 

In  the  above  equations,  [r]  indicates  the  maximum  Integer  not  exceeding  r.  Figure  2  shows 
relations  between  p  and  the  unit  locations.  The  remaining  connection  weights  are  all  zero. 
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3.2  La  and  Networks 
Connection  weights  are  defined  by. 

La  Net  bRK:  Bidirectional  connection  weight  between  u(i,j)  and  u(l+k,J+k). 

La  Net  bLK-'  Bidirectional  connection  weight  between  u(i,j)  and  u(i-k,j+k). 

The  input  of  u(i.j)  is  expressed  by 

La  Net:  x{l,j)  =  sy(i.j)  +  X  bRK[y(I-k.  j-k)  +  yCl+k.J+k)]  (7) 

k 

La  Net:  x(i,J)  =  sy(i,j)  +  I  b^klyd+k.  j-k)  ♦  y(i-k.J*k)]  (8) 

k 

The  same  conditions  can  be  derived  for  s.  bRk  and  buk  as  in  the  Li  network. 

3.3  Interaction  among  Li,  La.  La  and  L^  Networks 

In  Li  through  La  networks,  If  the  minimum  length  is  chosen  to  be  relatively  long,  curved 
lines  and  anotlier  Inclined  lines  cannot  be  extracted.  On  the  contrary,  if  the  minimum 
length  is  chosen  to  be  short,  many  non-line  segments  are  extracted.  In  order  to  avoid 
such  undesirable  extractions,  multistage  e.itraction  and  interaction  among  the  Li  through 
La  networks  are  employed. 

Stepl:  The  vertical  and  horizontal  lines  with  minimum  length  of  m-units  are  extracted,  m  is 
chosen  to  be  relatively  large.  The  extracted  line  segments  are  removed  from  the  original 
pattern.  The  remaining  pattern  is  set  on  the  Li  and  La  networks.  The  line  segments  with 
(m-l)-unit  lengths  are  extracted,  using  the  connection  weights,  which  satisfy  Eqs.(4)-(6). 
This  step  is  repeated  by  decreasing  the  lengths.  The  extracted  line  segments  are  combined 
resulting  the  final  vertical  and  horizontal  line  segments. 

Step2:  The  inclined  line  segments  (±45  degrees)  with  the  minimum  length  of  m-unIts  are  ex¬ 
tracted,  The  extracted  line  segments  are  removed  from  the  original  pattern.  By  setting 
the  remaining  pattern  on  the  La  and  La  networks,  the  same  line  extraction  with  (m-l)-unit 
lengths  is  repeated.  This  step  is  repeated  by  decreasing  the  lengths.  The  extracted  line 
segments  are  combined  resulting  the  final  Inclined  line  segments. 

Step3:  If  the  extracted  Inclined  line  segments  are  completely  included  in  the  previous 
vertical  or  horizontal  line  segments,  then  they  are  removed. 


IV  ANGLE.  MIDDLE  POINT  AND  LENGTH  EXTRACTION 
4.1  Angle  Extraction  in  A-Layer 

Deviation  from  the  standard  angles,  that  is  vertical,  horizontal  and  slopes  with  ±  45 
degrees,  is  detected  by  using  the  following  cellular  neural  network.  Connection  weights. 


defined  in  3.1  and  3.2,  are  determined  as  follows: 

Ai  Net  (vertical);  bRi  =  -1.  b^i  »  1  (9a) 

Aa  Net  (horizontal):  bRi  =  1,  bui  =  -1  (9b) 

As  Net  (+45  degrees):  avt  =  1.  ani  =  -1  (9c) 

A4  Net  (-45  degrees):  avi  =  -1  awi  =  1  (9d) 

The  unit  input  Is  expressed  as  follows: 

x(i.j)  =  avi[y{l,J+l)  +  y(l,J-l)]  ♦  aHi[y(i+l.j)  ♦  y(l-l.j)] 

♦  bR,[y(i+l.J+l)  +  y(l-l.J-l)]  +  bi,,[y(i+l.  j-l)  +  y(i-l,J+l)]  (10) 

The  angle  deviation  is  given  by 

°"'2(Nu-i)  (11) 


No  is  the  number  of  units  Included  In  the  line  segments.  Q  u  means  a  set  of  coordinates  of 
the  units.  D  represents  the  deviation  from  the  standard  angles.  The  standard  angles  are 
also  normalized  as  follows: 


Slope  (-45  degrees):  -1 
Horizontal  line:  0 

Slope  (+45  degrees):  1 
Vertical  line:  2 


Letting  the  normalized  angle  be  Ao.  the  v-hole  angle  A  is  calculated  as  follows: 

A  =  ((  Ao  +  D  ))a,  ((  n  ))n  means  n  modulo  N.  (12) 
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4.2  Middle  Point  and  Length  Extraction  in  TH-Laycr 

The  middle  point  and  the  length  of  the  line  segment  are  obtained  by  tracing  the  line 
segments,  starting  from  the  end  points.  The  connection  wclglits  are  determined  as  follows; 


w(i.j)  =  f  Wl^t.  i=j  (13) 

1  0,  i^^j 

av!  -  an  1  ”  bRi  =  bui  ■  1  (14) 

w(I,j)  is  connection  weight  from  the  ith  unit  In  L-layer  to  the  jth  unit  In  TR-layer.  The 
input  of  u(i,J)  Is  expressed  by 

x(i,J)  =  WcTypIiJ)  +  avi[y(l,j+l)+y(ij-l)]  ♦  aHi[y(l+l,J)+y{i-l, j)] 

+  bRt[y(i+l.j+l)+y(i-l,  j-D]  +  buifyd+l.  j-l)+y(i'l.  j+D]  (15) 

yr(i,j)  and  y(i.j)  are  the  outputs  of  u(l,J)  In  the  L-layer  and  the  TR-layer,  respectively. 
At  the  first  network  transition,  the  state  of  u(l,J)  In  the  TR-layer  Is  determined  by 

y(i,j)-M.  x(i,j)S0  (16) 

\  0,  otherwise 

At  the  following  steps,  the  output  Is  determined  by 

Ifx(i,j)2:0,  then  y(l,j)  =  x(i.j)  -  0  *2  (17) 


By  setting  0  =lVL-r+l,  the  end  points  of  the  line  segment  are  activated  at  the  first  step,  in 
the  following  steps,  the  neighborhood  of  the  activated  units  can  be  successively 
activated. 

The  middle  point  can  be  detected  as  the  colliding  point  of  two  traces.  Furthermore,  the 
length  of  the  line  segment  can  be  obtained  as  a  sum  of  the  outputs  of  the  colliding  units. 
The  length  is  normalized  by  the  total  number  of  units  included  In  the  original  pattern. 

V  PATTERN  RECOGNITION  BY  FEATURE  MAPPING 


5.1  Mental  Distortion 

In  the  proposed  method,  standard  pattern  Is  mapped  onto  the  distorted  version,  while 
maintaining  topological  structure.  Because,  the  former  is  familiar  to  the  human  brain. 
Therefore,  this  mapping  process  can  be  regarded  as  mental  distortion  In  human  brain. 

5.2  Candidate  of  Standard  Patterns 

Line  segment  features  of  standard  patterns  are  extracted,  and  are  stored  in  the  memo¬ 
ry.  Candidate  of  the  standard  patterns  are  selected  by  comparing  the  number  of  the  line 
segments  characterized  with  three  kinds  of  features. 

5.3  Structui'o  Tnv.-iriant  Feature  Mapping 

Kohonen's  self-organizing  map  [8]  Is  improved  as  follows: 

(1) Feature  points  of  the  standard  pattern  are  mapped  onto  those  of  the  distorted  pattern. 

(2) Feature  points  are  mapped  onto  the  corresponding  feature  points. 

(3)  Feature  points  are  selected  in  the  variable  ring  shape  region,  in  order  to  make  it  easy 
to  find  the  corresponding  feature. 

(4) Feature  points  are  selected  from  both  patterns,  in  order  to  avoid  double  mapping 
and  oscillation  in  the  mapping  process. 

The  proposed  feature  mapping  process  Is  described  in  the  following. 

(Stepl)  Selecting  Feature  Points: 

The  mapping  Is  carr'od  out  on  NxN  grids.  Feature  points  in  both  patterns  are  selected 
in  Che  1st  outside  region,  whose  coordinates  are  given  by  (1,J).  (N,j).  (1,1),  (l.N),  1,J=1~N. 
(Step2)  Selecting  Corresponding  Feature  Point: 

When  Pk  Is  selected  first,  q™,  which  satisfies 

a  i  Lk-Lm  i  ♦  /9  I  Ak-Am  |  <  c  ,  a  and  ^  are  weighting  factors  (18) 

is  selected  as  the  corresponding  feature  point.  Lk.  Ak  and  Lm.  Am  are  the  lengths  and  an¬ 
gles  of  Pk  and  dm.  respectively.  If  several  satisfy  this  condition,  one  of  them,  locates 
closest  to  Pk,  Is  finally  selected. 

Pk  is  shifted  toward  the  selected  qm-  On  the  other  hand,  when  q™  Is  selected  first,  Pn. 
which  satisfies  the  above  conditions,  is  selected  as  the  corresponding  feature  point.  In 
this  case,  Pk  is  also  shifted  toward  qm. 
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(steps)  Neighborhood  Constraints: 

When  Pk  Is  shifted  toward  q™.  It's  neighborhood  are  aiso  shifted  toward  the  same 
direction  with  shorter  distance  than  that  of  Pk- 
(Stcp4)  Narrowing  Ring  Shape  Region: 

Feature  points  are  selected  in  the  2nd  outside  region,  wliose  coordinates  are  given  by 
(2.J),  (N-l.J),  (i.2)  and  (l.N-1),  1,J=2~N-1.  Steps2  and  3  arc  repeated  for  all  feature  points 
in  this  region.  After  the  region  reaches  the  central  point,  the  mapping  process  returns  to 
Stepl.  The  above  processes  are  further  repeated  until  the  mapping  converges. 

5.4  Estimation  of  Napping  Results 
Similarity: 

The  feature  distribution  P  is  assumed  to  be  changed  to  P',  after  the  mapping.  In  order 
to  estimate  similarity  between  P'  and  Q.  the  following  error  function  Is  employed. 

c _ S[p’(n).q(m)l, _ 

I  [p’(n),q{m)]i  +  5:[p'(n)]2+  I  [q(m)]a  ^  ^ 

[]i:  The  number  of  pairs  of  p'  k  and  qm.  which  are  mapped  onto. 

(Is:  The  number  of  p’ k.  which  are  not  mapped  onto  the  corresponding  feature  point. 

[la:  The  numbers  of  q^.  onto  which  any  p’k  are  not  mapped. 

Therefore,  perfect  mapping  yields  S=l,  otherwise  S<1. 

Convergence  Rate: 

Feature  point  mapping  from  the  1st  region  to  the  central  region  is  regarded  as  one 
iteration.  A  convergence  rate  is  measured  by  the  number  of  this  iteration. 

Variance  from  Standard  Pattern: 

Relative  deviation  between  P’  and  P  Is  estimated.  Let  (l,j)  and  (I’.j’)  be  the  coordinates 
.of  elements  in  P  and  P',  respectively.  Average  of  translation  (im,jj»)  is  given  by 

im~X(l'-i).  (20) 

where  Nl.  is  the  number  of  the  line  segments.  The  relative  distortion  is  estimated  by 

V,  ~S(i'-l-im)=.  Vj  =-il(j’-j-j„)*  (21) 

VI  SIMULATION 

6.1  Japanese  Kanji  Characters 

Handwritten  Japanese  Kanji  characters  have  been  dealt  with.  They  are  expressed  using 
two- level  values  and  24  x  24  dots.  The  first  group  of  JIS  Kanji  characters  (2965)  are  used 
for  standard  patterns. 

6.2  Handwritten  Kanji  Character  Recognition 
Figure  3  shows  an  example  of  a  handwritten 

distorted  pattern,  and  it's  skeletonized 
pattern.  Figure  4  shows  the  extracted  line 
segments  in  the  L-layer.  The  minimum  line 
length  in  the  first  step  is  chosen  to  be  m=3, 
and  the  line  extraction  is  repeated  using 
m-l=2.  Since  some  margin  is  employed  for  se¬ 
lecting  the  standard  patterns,  and 
topological  structure  Is  not  taken  Into  ac¬ 
count,  16  different  Kanji  characters  are  Flg.3  Example  of  distorted  pattern  . 
selected.  Among  them,  fSj  ,  TSJ  ,  f ?E j  ,  T  (a)  input  pattern 

£J,r(£J  and  fPj  have  similar  structure  (b)  Skeletonized  pattern, 

to  that  of  the  distorted  pattern. 

Figure  5  shows  feature  point  distributions  of  the  standard  pattern  and  the 

distorted  version  ■,  and  shifting  directions. 

The  three  measures,  obtained  by  the  feature  mapping,  are  listed  in  Table  1.  Since  , 

rSJ  ,  rSj  have  almost  same  topological  structure,  the  similarities  defined  by  Eq.(19) 
become  S=l.  Therefore,  they  cannot  be  distinguished  based  only  on  the  similarity. 
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Fig  5  Feature  point 
distributions,  and 
shifting  directions. 


Table  1  Three  measures  and  order  of  standard  patterns, 
estimated  by  feature  mapping. 


Standard 

Patterns 

Similarity 

S 

I teration 

Variance 

Vi  Vi 

Order  of 

Estimation 

I.  00 

12 

0.69,  3.39 

1 

S 

1.00 

15 

0.82,  6.78 

2 

1.00 

16 

6.82,  6.24 

3 

0.92 

13 

10.2,  4.14 

4 

0.  92 

13 

8.22,  16.0 

5 

F 

0.83 

11 

0.56,  2.64 

6 

VI  CONCLUSIONS 

A  new  approach  to  handwritten  Japanese  Kanji  character  recognition  has  been  proposed. 
The  ideas  behind  the  proposed  are  based  on  line  feature  extraction  by  the  cellular  neural 
networks  and  mental  distortion  by  the  structure  invariant  feature  mapping.  Computer 
simulation  using  about  3000  Kanji  characters  has  demonstrated.  Distorted  patterns  with 
scaling,  translation,  rotation  and  general  distortion,  can  be  recognized. 
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Abstract 


The  idea  of  nonlinear  and  delay-type  clonning  templates  whidi  enable  to  increase  essentially  the 
^ciency  of  the  CNN  was  described  in  [1].  This  paper  is  devoted  to  the  image  thiddng  templates  whidi  enable 
to  improve  the  quality  of  distored  images.  Espedally  it  was  found  that  these  type  of  the  CNN  can  be  successfully 
used  in  preprocessing  of  handwritten  characters  in  order  to  recognize  them  by  the  four  COO  operators  method 
[4].  Three  various  thiddng  templates  are  considered  (demonstrated)  :  two  examples  of  nonlinear  clonning 
templates  and  one  delay-type  template. 


1.  Introduction 

The  cellular  neural  networks  (CNN)  are  effective  tools  for  image  processing  due  to 
their  simple  operation  mode  and  regular  configuration  [2,3].  The  number  of  the  CNN 
applications  has  been  rapidly  growing.  It  was  shown  that  these  networks  can  be  efficiently 
used  for  the  handwritten  character  recognition  by  the  connected  component  detectors  (CCD) 
method  [4].  It  is  observed  that  primary  images  of  handwritten  characters  obtained  from  the 
scanner  have  irregular  noisy  contours.  This  fact  may  decrease  the  efficiency  of  the 
recognition  by  the  CCD  method. 

The  contours  of  figures  can  be  improved  (i.e.  smoothed,  regularized)  by  thicking. 
Three  various  CNN  algorithms  of  thicking  are  presented  in  this  paper: 

•  nonlinear  clonning  template  I  -  simple  thicking; 

•  nonlinear  clonning  template  n  -  which  enables  to  obtain  the  figure  contour  and 
additionally  thicking  by  one  pixel  after  logical  "OR"  operation  with  the  input  image; 

•  delay-type  template  which  gives  efficient  smoothing  and  thicking  by  local  averaging. 


0-7803-0875-1/92  3$.00  ©1992IEEE 
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2.  Thicking  donning  templates 


2.1.  The  nonlinear  template  I  for  contour  thicking  may  be  expressed  as  follows  using  the 
usual  CNN  notation: 


1  d  1 

1  1  1 

A  = 

d  5  d 

B  = 

1  -1  1 

1  d  1 

1  1  1 

(1) 


where  d  =  0.5  •  (  Y  [  k  ,  1  ] )  ^  -  (  Y  [  k  ,  1  ]  •  Y  [  i  ,  j  ] )  . 


Due  to  this  rule  the  contour  of  an  object  can  be  increased  by  one  cell  in  all  directions,  as 
shown  in  Fig.  1.  It  does  not  require  any  special  initial  and  boundary  conditions  -  it  is 
sufficient  to  impose  -1  at  the  boundary  and  to  take  the  image  itself  as  an  initial  condition. 
The  operation  mode  takes  only  several  iterations  while  simulating  on  a  computer. 


(a) 


(b) 


(c)  (d) 

Fig.  1:  Demonstration  of  the  thicking  template.  (a)(b)(c)(d)  Input  and  ou^ut  images. 
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As  can  be  seen  from  Fig.l.  the  donning  template  (1)  improves  the  contours  of  input 
images  by  smoothing.  This  rule  is  characterized  by  good  stability  properties. 


2.2  The  nonlinear  donning  template  11  for  drawing  contours  is  described  below; 


'0  d,  0  ■ 

4  4  4 

A  = 

da  0  d, 

B  = 

4  -30  4 

0  di  0 

4  4  4 

where  di 
^2 


(  Y  [  k  ,  1  ]  )  2  -  0.5  •  (  Y  [  k  ,  1  ]  .  Y  [  i  ,  j  ]  )  , 

-  0.5  •  (  Y  [  i  ,  j  ]  ♦  Y  [  k  ,  1  ]  )  . 


(2) 


This  rule  enables  to  draw  the  contour  of  a  given  object  in  all  directions  and  the 
contour  is  shifted  by  one,  as  shown  in  Fig.  2.  The  operation  mode  takes  few  iterations  (  or 
time  costants  in  analog  circuit).It  is  possible  to  use  this  rule  many  times  to  obtain  multiple 
contours  of  figures. 


(c)  (d) 

Fig.  2:  Demonstration  of  the  drawing  contour  template.  (a)(b)(c)(d)  Input  and  output  images 


In  order  to  obtain  the  image  thicker  by  1  pixel  it  is  necessary  to  perform  the  logical 
"OR"  of  the  computed  contour  and  the  input  image.  This  operation  can  expressed  by: 

A  *  1  B  =  1  I  =  1  (3) 

The  examples  of  results  are  illustrated  in  Fig.  3. 
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(a)  (b)  (c)  (d) 

Fig.  3:  Demonstration  of  the  thicking  through  drawing  contour  and  logical  "OR"  with  input 
image.  (a)(b)(c)(d)  Output  images. 

23  Delay-type  donning  template  for  improvement  of  handwritten  character  image  at  the 
output  of  a  scanner  has  the  following  form: 


0 

0.25 

0 

A  = 

0.25 

0 

0.25 

o 

II 

I  =  0.0 
tl  =  6 

0 

0.25 

0 

A'’  = 

1 

B“  = 

0 

1“ 

=  0.5 

t2  =  9 

A*^  = 

3 

B‘2  = 

0 

=  1.5 

The  template  improves  the  quality  of  the  character  by  noise  removing  and  by 
smoothing  the  irregularities  at  the  border,  as  presented  in  Fig.  4.  The  image  itself  is  used  as 
the  initial  condition  and  all  boundary  cell  states  are  equal  to  -1. 


(c)  (d) 

Fig.  4:  Demonstration  improving  quality  of  handwritten  characters.  (a)(b)(c)(d)  Input  and 
output  images  after  once  use  of  template  (3). 
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This  template  operates  in  three  steps. 

-  STEP  I  :  six  iterations  of  the  rule  determined  by  A,  B  and  1; 

-  STEP  II  :  three  iterations  defined  by  the  operators 

A’=A  +  A“  B’=o  r=i+r' 

-  STEP  III:  a  sequence  of  iterations  obtained  in  the  following  way: 

A"=  A’+A*^  B"=  0  r=r+i^ 


This  donning  template  may  be  modified  by  changing  the  template  parameters:  the 
delays  tl  and  t2  and  current  values.  The  STEP  I  of  the  algorithm  causes  the  averaging  of  the 
input  image  -  the  contour  becomes  fuzzy  and  the  networks  tends  to  the  grey-scale  state.  This 
effect  increases  in  function  of  time.  The  averaging  stops  by  using  the  STEP  II  -  the  network 
tends  to  the  stable  state  -1  (white  image)  in  the  whole  area  of  the  considered  image.  Then 
the  use  of  the  STEP  III  makes  the  stable  output,  thicker  than  the  input  and  with  the  smooth 
contour.  Increasing  of  the  delays  tl  and  t2  sunpliBes  the  thicking  effect  (up  to  hole-filling), 
as  can  be  stated  from  Fig.  5. 


(a)  (b)  (c)  (d) 

Fig.  5;  Improving  quality  of  handwritten  characters,  (a)  Input  image,  (b)  Output  image  : 
tl=6,t2=9.  (c)  Output  image  :  tl=8,t2=13.  (d)  Ouq)ut  image  :  tl  =  ll,t2=32. 


3.  Conclusions 

Three  new  donning  templates  for  the  CNN  are  described  in  this  paper.  Their  aim  is 
to  perform  the  improvement  of  noisy  image  with  highly  irregular  contour  by  the  image 
thicking  and  contour  smoothing.  These  operations  cause  the  regularization  of  hand-written 
character  images  after  scanning  and  make  character  recognition  more  efficient  by  using,  e.g. 
the  connected  component  detector  method.  All  presented  templates:  nonlinear  I  and  II  as  well 
as  the  double-delay-type  template  are  stable  and  useful.  The  simplest  is  the  nonlinear  I 
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template.  The  nonlinear  II  template  consists  in  obtaining  the  outer  contour  of  the  figure  and 
thicking  by  the  "OR"  operation  with  the  input  image.  The  delay-type  operation  can  be 
modified  by  changing  its  delay  parameters.  The  results  of  operation  of  presented  templates 
are  confronted  in  Fig.  6. 


(a)  (b)  (c)  (d) 

Fig.  7:  Comparision  of  input  and  output  images  after  thicking  by  following  templates. 

(a)  Input  hand-written  character,  (b)  Thicking  by  nonlinear  template  I.  (c)  Thicking  by 
nonlinear  template  O.  (d)  Thicking  by  delay-type  template. 
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Abstract 

Cellular  neural  networks  (CNNs)  are  considered  here  as  cellular  analog  programmable 
multidimensional  processing  arrays  with  distributed  logic  and  memory.  The  interconnecting  weights 
between  the  neighbouring  processing  elements  are  defined  by  the  template  values.  In  the  paper  a  new 
systematic  way  is  presented  to  find  robust  templates.  Using  the  new  learning  algorithm  some  templates 
were  found  for  a  CNN  based  layout  design  rule  checking  algorithm.  The  algorithm  has  been  tested  in 
our  experimental  system  with  real  life  examples.  A  typical  design  rule  checking  of  a  432x164  pixels  area 
takes  8s  computation  time  on  our  CNN  hardware  accelerator  board. 

1  introduction 

Cellular  neural  networks  [1]  (CNNs)  are  considered  here  as  cellular  analog  programmable 
multidimensional  processing  arrays  with  distributed  logic  and  memory  [11], 

The  interconnecting  weights  between  the  neighbouring  processing  elements,  cells,  are  defined 
by  cloning  template  values.  Besides  the  linear  cloning  templates,  the  nonlinear  and  delay  type  templates 
proved  to  be  substantially  widening  the  applications  (3).  A  cloning  template  defines  the  transformation  of 
an  input-  and  initial  state  signal-array  (e.g.  image)  to  an  output  signal-array.  Some  templates  have  been 
found  by  a  “cut-and-try"  method,  but  recently,  systematic  template  learning  algorithms  were  reported 
[5], [7], (10], [13].  The  learning  process  presented  here  is  based  on  simplex  optimization  method.  The 
learned  templates  are  robust,  i.e.  they  are  less  sensitive  to  the  changing  of  the  values  of  their  elements. 

The  layout  rui>j  checking  of  a  printed  circuit  board  documentation  or  a  manufactured  board  is 
time  consuming.  Detection  of  layout  errors  requires  mostly  local  geometrical  information,  hence  the  task 
well  suits  the  CNN  paradigm.  The  typical  layout  errors  on  a  PCB  artwork  film  or  on  a  manufactured  PCB 
are  as  follows:  (i)  wire  width  is  smaller  than  a  given  value,  even  a  wire  may  be  broken,  (ii)  the  isolation  on 
the  layout  is  smaller  than  a  given  value,  even  a  short  circuit  may  be  produced,  (iii)  the  brake  of  a  pad,  (iv) 
fleck  or  pinhole  on  a  wire,  and  (v)  the  misalignment  of  the  pads  to  the  holes.  A  key  problem  is  the 
detection  of  the  minimal  line  width  violations.  It  can  be  shown  that  most  of  the  layout  error  detections  can 
be  transformed  to  this  problem. 

Considering  the  different  CNN  hardware  implementations  [2], [6]  it  can  be  seen  that  for  large 
images  (over  100  thousand  pixels)  the  digital  multiprocessor  add-on-board  is  the  only  solution.  Our  new 
CNN-HACM  [4]  board  provides  a  1  million  cell  space  at  1 .5  /is/cell/iteration  speed.  Here,  a  low  cost 
experimental  solution  will  be  shown  based  on  our  CNN-HACM  hardware  accelerator  board  to  solve  the 
layout  design  rule  checking  problem.  The  most  attractive  solution  is,  naturally,  the  analog  VLSI  solution. 
An  IBM  add-on-board  with  a  0.4mm  minim'll  wire  width  (with  200  dot/inch  resolution)  can  be  processed 
in  2  parts  using  our  new  CNN-HACM  accelerator  board. 

Template  learning  methods  are  introduced  with  novel  properties  in  Section  2.  New  templates 
and  subroutines  [13]  are  developed  in  Section  3  to  solve  the  above  basic  layout  detection  problem.  Real- 
life  examples  are  summarized  in  Section  4.  Finally,  in  Section  5  the  theoretical  and  the  practical 
limitations  are  discussed. 
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2  Learning  algorithm  for  CNN 

Template  design  or  learning  algorithm  for  cellular  neural  networks  means  a  procedure  which  is 
able  to  find  the  template  for  a  given  operation.  The  operation  is  specified  by  the  two  input  signal  arrays 
Si  (ij)  =  Vuij  and  S2(ij)  =  Vj^jj(0)  and  the  desired  output  S^Oj) =Vyjj(y). 

Learning  algorithms  published  so  far  [5], [7], [10). [13]  are  based  on  a  system  of  inequalities  and 
by  solving  them,  linear  templates  are  obtained.  Consider  the  state  equation  in  [l.a].  Assuming  that  the 
equilibrium  points  of  the  network  are  stable.  i.e.  for  t-«>  dv  -  •/dt=0,  it  follows  that 

X.1J 

V  .  .  (<x> )  =  y  A  ■  ■  ij*v  ,,  +  Y  B  •  ■  ii*v  ti  +  I  •  ■ 

XI j'  ^  ^  ij:kl  ykl  ^  ij'.kl  ukl  ij 

kl  G  N^(ij)  kl  G  N^dj) 


where  Is  the  settled  state.  Assuming  further  that  the  desired  output  is  binary,  i.e.  v  jTt) =±1  for 
and  settirig  Vyjj(“)  to  the  desired  output  S^Oj),  because  of  [1  .a],  we  get 

^  ij;kl  ykl  ^  ij;kl  ukl  ij  -  ^  y^y  ^ 

kl  G  N/ij)  kl  G  N^djJ 

Y  A  ■  ■  ii*v  It  +  y  B  ■  ■  fi*v  11  +  I  ^  - 1  if  V  •  •(“ 1 . 

^  ij;kl  ykl  ^  ij;kl  ukl  ij  Y^J 

kl  G  N^dj)  kl  G  N^dj) 


To  assure  stability  the  symmetry  of  the  feedback  template  A  is  provided.  To  gain  a  more 
efficient  template  the  following  inequalities  can  be  added  to  the  above  system  providing  a  shorter  and 
possibly  monotone  transient: 

y  A  ■  ■  ii*v  t  1  (0)  +  Y  B  •  ■  ii*v  11  +  I  ■  ■  -V  ••(0)3:  0 

^  ij;kl  ykl'-  '  ^  ij;kl  ukl  xj  xiy 

kl  G  N^dj)  kl  G  N^dj) 

if  Vy^jfoo)”!  and 

y  A  •  ■  I  t*  V  I  I  (0)  +  Y  B  ■  ■  ii*v  II  +  I  •  •  ~v  ••(0)5  0 

I-  ij;kl  ykl'-  '  ^  ij;kl  ukl  ij  xiy 

kl  G  N^dj)  kl  G  N^dj) 

if  Vy£j(“)=— 1  and  Vj^£j(0)>-1. 

In  [5]  relaxation  methods  were  used  to  find  a  solution  of  the  system  of  inequalities.  Some 
useful  templates  had  been  found,  but  in  many  cases  changing  the  template  elements  by  a  few  percent, 
the  dynamic  behaviour  of  the  network  changed  as  well  and  the  template  no  longer  performed  the  desired 
operation.  To  overcome  this  difficulty,  a  different  algorithm  was  proposed  [7]  which  is  discussed  in  short 
here. 


A  learning  algorithm  for  gaining  robust  templates 

To  obtain  robust  templates,  we  have  to  find  not  only  an  arbitrary  solution  of  the  Inequality 
system,  but  to  find  an  optimal  one  in  the  sense  of  being  far  inside  the  solution  domain.  The  robustness  erf 
a  template  with  respect  to  changing  its  elements  depends  on  how  far  inside  the  parameter  space  it  is 
from  the  boundaries  defined  by  the  inequalities.  The  greater  the  distance,  the  more  robust  the  template, 
and  the  transient  gets  faster,  as  well. 

To  characterize  a  point  in  the  parameter  space  a  cost  function  is  constructed  whose  global 
minimum  is  as  far  as  possible  from  all  of  the  boundaries  of  the  solution  domain.  Locating  the  minimum  of 
the  cost  function  by  some  optimization  method,  a  template  with  the  desired  properties  can  be  gained. 

To  find  this  optimum  the  simplex  optimization  algorithm  has  been  applied.  If  the  domain  of 
solutions  in  the  (2r-i- dimensional  parameter  space  (the  number  of  template  elements  with 
neighbourhood  size  r)  is  closed,  then  the  "mass  centre  point"  of  the  domain  should  give  a  robust 
template.  In  that  case  the  cost  function  is  defined  as 
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cost (T) 


2  <5(T,p-) 


where  cSCT.p^)  denotes  the  signed  distance  between  the  point  in  the  parameter  space  corresponding 
to  the  parameter  vector  (or  template)  T  and  the  plain  defined  by  the  /th  inequality.  It  is  easy  to  see  that 
this  cost  function  is  unimodal.  that  is.  the  optimization  algorithm  finds  the  global  minimum  without 
difficulty.  There  is  no  guarantee,  however,  that  the  domain  of  solutions  is  closed.  In  fact,  in  most  cases 
the  solution  domain  is  open  and  the  optimum  lies  in  the  infinity.  To  overcome  this  problem  and  to  provide 
a  controllable  robustness  for  the  templates,  the  following  learning  method  has  been  used. 

1 .  First,  the  simplex  algorithm,  which  works  with  a  set  of  parameter  vectors,  is  initialized  with 
templates  having  symmetric  feedback  A.  The  algorithm  then  guarantees  that  all  parameter  vectors 
remain  in  the  subspace  defined  by  the  symmetry  constraints. 

2.  In  the  first  optimization  step  the  simplex  algorithm  is  run  to  bring  all  parameter  vectors  inside 
the,  possibly  open,  solution  domain.  It  stops  when  all  the  points  corresponding  to  the  parameter  vectors 
are  at  least  e  >  0  far  from  the  boundaries. 

3.  In  the  second  run  a  different  cost  function  is  applied  to  bring  the  parameters  into  the 
feasibility  range  while  the  condition  acquired  in  step  2  is  maintained. 

This  method  gives  templates  with  a  prescribed  robustness  with  respect  to  changing  their 
parameters.  This  means  that  the  template  elements  can  be  slightly  modified  without  changing  the 
behaviour  of  the  system.  Alteration  of  the  elements  has  no  effect  on  the  operation  performed  by  the 
template  at  least  until  the  absolute  sum  of  the  modifications  remains  under  e. 

Limitations  of  the  proposed  learning  algorithm 

Learning  algorithms  based  on  the  above  system  of  inequalities  have  strict  limitations.  Only 
binary  output  templates  with  symmetric  feedback  can  be  generated.  Besides,  no  information  about  the 
dynamics  of  the  actual  transient  is  considered.  Consequently,  templates  with  local  dynamics  can  be 
learned  only. 

Recently  a  new  learning  algorithm  [14]  based  on  genetic  search  was  reported.  According  to 
the  experiences  a  broader  class  of  template  learning  problem  can  be  solved  by  using  this  algorithm. 

3.  The  experimental  system  for  PCB  layout  error  detection 

The  main  steps  of  PCB  layout  error  detection  are  as  follows, 

(i)  the  optical  input  of  the  layout  (in  our  experimental  system  we  use  a  scanner  input), 

(ii)  the  transformation  of  the  gray  level  image  of  layout  to  a  black  and  white  one, 

(iii)  perform  the  different  layout  error  detection  algorithms. 

3.1  Setting  the  appropriate  parameters  of  the  optical  scanner 

Using  an  Epson  GT4000  scanner  [9],  scanning  was  tested  the  typical  classes  of  printed  circuit 
boards  and  PCB  artwork  films.  The  point  here  is  to  have  such  a  scanned  gray  scale  area  where  the 
layout  figures  can  easily  be  identified  in  the  background  isolation  area.  The  resolution  of  the  scanning 
should  be  at  least  3-4  times  finer  than  the  smallest  details  of  the  scanned  object. 

3.2  The  black  and  white  image  generation 

The  next  step  is  to  find  a  black  and  white  picture  from  the  gray  scale  one.  Here  the  average, 
the  thresholding,  or  some  nonlinear  filtering  templates  can  be  used. 
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3.3  The  critical  algorithms  of  the  design  rule  checking  of  printed  circuit  board  documentation 
by  using  cellular  neural  networks 

A  crucial  problem  of  the  layout  checking  means  the  detection  of  violations  of  the  minimal  line 
width  and  the  minimal  separation  width.  The  second  problem  can  be  transformed  to  the  first  one.  The 
pixels  of  lines  or  pads  are  considered  black  and  the  area  between  them  white.  If  we  were  able  to  solve 
the  problem  of  minimal  line  width  violation  detection  on  a  positive  picture  then  on  a  negative  picture,  the 
minimal  separation  violation  could  be  detected  by  the  same  method. 

The  minimal  line  width  violation  detection  of  rectilinear  layout 

Here  a  sequence  of  image  transformations  will  be  shown  to  determine  the  places  where  the 
line  width  requirement  were  not  met.  The  sequence  of  image  transformations  in  CNN  can  be  done  by  a 
sequence  of  templates. 

The  horizontal  and  the  vertical  line  width  checkings  are  performed  separately  and  the  faults  are 
logically  OR-ed.  The  minimal  line  width  is  first  supposed  to  be  2  pixels,  and  later  an  algorithm  Is  shown 
for  N  pixel  minimal  line  width. 

The  vertical  wire  case 

First,  let  us  suppose  that  we  have  2  pixels  wide  lines  and  these  lines  in  some  cases  are  thinner 
than  2  pixels.  We  have  to  find  these  places.  Using  our  template  learning  program  [7J  with  e=  1.0,  the 
result  is  as  follows. 


-0.11 

0.43  -0.11 

0.19  -0.04 

0.19 

A= 

0.43 

0.06  0.43 

B= 

-2.92  2.40 

-2.92 

-0.11 

0.43  -0.11 

0.19  -0.04 

0.19 

. 

- 

The  template  finds  all  pixels  which  are  on  a  vertical  line  with  one  pixel  width.  Knovtring  that  this 
template  has  been  found  with  e=  1.0  value,  the  learned  template  could  be  rounded  as  follows: 


-0.1 

0.4  -0.1 

0.2 

0.0  0.2 

A= 

0.4 

-0.1 

0.0  0.4 

0.4  -0.1 

-3.0 

0.2 

2.5  -3.0 

0.0  0.2 

The  horizontal  wire  case 

To  detect  the  errors  in  horizontal  wires,  the  same  template  can  be  used  as  in  the  vertical  case, 
however,  the  template  B  has  to  be  rotated  by  90  degrees. 

The  general  minimal  line  width  violation  detection 

The  next  problem  is  to  find  the  places  where  a  line  is  thinner  than  N  pixels.  To  solve  this 
problem,  we  have  to  find  some  algorithms  containing  several  templates  (CNN  subroutines).  The 
algorithm  uses  peeling  templates  which  remove  pixels  from  either  sides  of  black  objects.  The  following 
template  turns  white  all  black  pixels  having  white  pixels  on  their  left  side: 


0  0  0 

0  0  0 

A  - 

0  2  0 

B  - 

3  3  0 

0  0  0 

0  0  0 

Templates  for  peeling  from  right  side  can  be  generated  by  rotating  the  template  B  above. 

The  CNN  subroutine  [13]  for  finding  lines  thinner  than  N  pixels  in  vertical  (horizontal)  direction 
is  as  follows.  Two  images  are  used,  namely  a  temporary  TEMP"  and  a  result  "RESULF. 
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-  Load  the  input  picture  and  save  it  in  TEMP 

-  Find  one  pixel  thin  lines  and  save  the  result  in  RESULT 

-  Repeat  N-2  times; 

-  Load  TEMP 

-  Peel  1  pixel  from  left  (up)  or  right  (down)  (best  if 
alternating) 

-  Save  in  TEMP 

-  Find  one  pixel  thin  lines  and  OR  the  result  in  RESULT 


Remarks: 

(i)  This  algorithm  has  to  be  run  twice  to  find  both  the  horizontal  and  vertical  errors. 

(ii)  It  can  be  seen  that  the  number  of  iterations  depends  on  the  minimal  wire  width  to  be  detected. 

(iii)  The  input  picture  and  the  initial  state  are  supposed  to  be  equal. 

(iv)  Here  an  additional  picture  has  to  be  used  to  store  the  results  (RESULT).  It  is  crucial,  in  view  of  speed, 
how  many  picture  changes  are  needed  and  how  many  additional  pictures  have  to  be  stored.  These  are 
the  parameters  which  determine  the  complexity  of  an  analog  or  a  dual  algorithm. 

After  error  detection  phase  the  very  short  (<d)  errors  should  be  removed,  supposing  they  are 
not  layout  errors  but  scanning  errors.  By  the  next  template  the  isolated  black  points  are  removed. 


A  = 

0  0  0 

0  2  0 

B  = 

0  0.5  0 

0.5  0.5  0.5 

0  0  0 

0  0.5  0 

4.  Layout  error  detection  experiences  based  on  nontrivial  examples 

Experiment  1 

A  part  of  an  artwork  film  of  a  multilayer  PCB  with  0.254mm  minimal  separation  width  was 
considered  and  the  places  have  to  be  detected  and  recorded  where  the  separation  is  smaller  then  this 
given  value.  The  size  of  the  picture  was  362*648  pixels.  In  Figure  1a  and  Figure  1b  the  scanned  binary 
input  and  the  detection  the  errors  of  vertical  separation  can  be  seen.  It  took  8s  processing  time  (and  an 
additional  20s  were  necessary  to  load  the  input  picture  and  the  initial  states)  by  using  our  CNN-HAC  (v. 
2.0)  board  [4]. 

Experiment  2 

The  gray  level  input  of  a  part  of  a  manufactured  printed  circuit  board  (603x309  pixels)  is  shown 
in  Figure  2a.  First,  a  black-and-white  picture  was  generated  (see  Figure  2b)  by  using  a  nonlinear  filtering 
and  the  average  template.  The  minimal  vertical  line  width  violation  (where  the  line  width  is  smaller  then  3 
pixels)  is  detected.  The  result  of  the  detection  is  shown  in  Figure  2c.  The  algorithm  was  used  and  the 
processing  time  with  the  CNN-HAC  board  [4]  was  10s  (an  additional  50s  were  necessary  to  load  the 
input  and  the  initial  states  and  to  draw  the  results  on  the  screen). 

5.  Theoretical  and  practical  limitations 

An  important  question  is;  in  how  many  parts  a  real  problem  can  be  processed.  Its  running  time 
consequence  was  shown  previously.  An  IBM  PC  add-on-board  with  a  0.4mm  minimal  wire  width  ( in  200 
dot/inch  resolution)  can  be  processed  in  2  parts  by  using  our  new  CNN-HACM  accelerator  board. 

With  the  proposed  method  the  short  circuits  cannot  be  detected  owing  to  the  fact  that  it  is  not 
a  local  error.  Even  if  we  could  give  a  node  information  to  a  pixel,  it  needs  an  additional  storage 
requirement  which  can  be  8  bits/pixel  or  more  Only  the  thin  short  circuits  are  detected  by  the  minimal 
line  width  violation  algorithm. 
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Abstract 

We  arc  studying  one  class  of  new  Cellular  Automata  (called  B-rule  in 
this  paper).  We  have  discovered  rule  P  can  he  exploited  in  the  design  of 
Cellular  Neural  Network  (CNN)  for  •  inary  image  thinning.  In  this  paper,  we 
introduce  B-rule  132  and  th;  iiiethod  of  the  binary  image  thinning  with  CNN. 

I  ntruduction 

Cellular  Neural  Network  (CNN)  [1-2]  is  a  class  of  novel  analog 
nonlinear  circuit  which  allows  real-time  signal  processing  by  using  only 
local  interconnections  of  neighboring  cells.  In  this  paper,  we  consider  a 
class  of  new  Cellular  Automata  (CA)  which  is  called  B-rule.  We  have 
discovered  that  the  rule  B132  can  be  exploited  in  the  design  of  CNN  for 
binary  image  thinning.  In  this  context,  the  characteristics  of  one¬ 
dimensional  (1-D)  B-rule  CA  is  introduced.  Then  we  discuss  the  approach 
and  results  of  employing  these  rules  to  binary  image  thinning  by  using 
CNN. 

The  relations  between  CA  and  CNN  linear  cloning  templates  have 
been  addressed  in  [3].  In  [4],  the  relations  among  CNN  with  CA  and 
Systolic  Array  (SA)  have  been  analyzed.  If  the  operations  of  the  system 
are  purely  logical  and  involve  only  a  few  bits,  the  classical  CA  of  John  Von 
Neuman  and  its  recent  variants  are  ideal  tools.  The  thinning  algorithms 
by  using  two-dimensional  (2-D)  CA  have  been  discussed  rigorously  in  [5] 
while  in  [6],  the  system  proposed  needs  a  complex  eight-plane  approach 
of  CNN  to  accomplish.  However,  the  authors  in  [6]  have  not  mentioned  the 
methodology  to  determine  the  values  of  the  cloning  templates.  In  the 
studies  of  a  new  class  of  1-D  CA,  we  have  discovered  that  rule  B]32  (to  be 
described  later)  are  useful  for  the  design  of  CNN  for  image  processing. 
Henceforth,  we  shall  refer  1-D  CA  simply  as  CA. 

B-rule  Cellular  Automata 

CA  are  sufficiently  simple  to  allow  detailed  mathematical  analysis, 
while  on  the  other  hand,  complex  enough  to  exhibit  a  wide  variety  of 
complicated  phenomenon.  In  a  simple  case,  a  CA  consists  of  a  line  of  cells 
or  sites,  each  with  value  of  0  or  1.  These  values  are  updated  in  sequence 
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of  discrete  time  steps,  according  to  a  definite  fixed  rule.  Denoting  the 

value  of  a  site  at  position  i  in  (r+l)th  time  step  by  ,  a  simple  rule  gives 
its  next  value  as 

Here  f  is  a  Boolean  function  which  specifies  the  rule,  a' j,  a-,  are  the 
values  of  the  cell  itself  and  the  immediately  adjacent  cells  in  the  tth  time 
step.  Tliis  rule  is  called  A-rule  CA  in  this  paper.  Thus,  there  are 
altogether  2^  =  256  possible  distinct  CA  rules  in  one-dimension  CA  with 
three  variables.  S.  Wolfram  [7]  has  studied  the  special  properties  of  this 
rules  and  has  discovered  that  there  exists  only  32  "legal"  rules.  Following 
[7],  a  rule  is  said  to  be  "legal"  if  it  is  (i)  reflection  symmetric,  and  (ii)  if 
given  an  initial  state  of  all  Os,  the  state  remains  unchanged.  Now, 
consider  a  class  of  CA  with  the  following  activation  rule: 

Here  cP  is  a  Boolean  function  which  specifies  the  characteristics  of  the  rule. 

The  value  of  depends  on  the  values  of  (at  the  rth  time  step) 

and  a/"'  (at  the  (r-l)th  time  step).  This  rule  is  called  B-rule  CA  in  this 
paper.  Thus  A-rule  is  equivalent  to  the  Ist-order  partial  differential 
equations  while  B-rule  the  2nd-order  partial  differential  equations 
relating  the  input  state  variables.  Likewise,  we  have  discovered  32 
"legal"  rules  for  B-rule  CA  while  the  evolution  of  the  patterns  can  further 
be  partitioned  to  5  classes  as  in  Table  1.  Moreover,  only  class  2  has  the 
potential  to  be  applied  to  image  processing.  In  particular,  rule  B]32  can 
be  applied  to  binary  image  thinning.  Fig.  1  shows  some  examples  of  the 
evolution  of  rule  B132  with  different  initial  conditions. 

Binary  Thinning  with  B-rule  CA 

Thinning  has  found  a  wide  application  in  the  field  of  character 
recognition  in  which,  prior  to  encoding  of  a  character,  it  is  necessary  to 
reduce  the  original  image  to  simple  lines  [5].  Thinning  is  actually  much 
more  difficult  than  it  looks,  basically,  two  tasks  must  be  implemented:  (i) 
peeling  the  thick  pixels  off,  and  (ii)  stopping  the  peeling  process  when  the 
pixel  size  is  exactly  one  or  two.  Our  proposed  algorithm  B-rule  CA  can 
implement  the  above  two  tasks  at  the  same  time. 

In  this  paper,  we  transform  the  image  thinning  process  from  a  2-D 
problem  into  two  1-D  CA  problems  (verticaLY  and  horizontal:X).  In 
accordance  with  the  evolution  characteristics  of  B]32  CA,  any  odd  number 
of  Ts  present  at  the  initial  state  leads  to  one-pixel  thick  stable  state.  Fig. 
2  shows  the  schematic  of  the  partitioned  network.  2-D  CA  is  separated 
into  two  1-D  CAs  nominated  with  X-CA  and  Y-CA.  To  ensure  the  correct 
operations  for  the  cells  on  the  boundary,  an  "zero  boundary  condition"  is 
assumed. 

Fig.  3  shows  some  simulation  results  of  a  40x40  pixels  printed 
character.  In  Fig.  3a,  the  original  character  is  presented,  where  the 
outputs  by  using  Y-CA  and  X-CA  are  respectively,  shown  in  Fig.  3b  and  3c. 
Finally,  combining  the  results  at  X  and  Y  directions  give  Fig.  3d.  Fig.  4  is 
the  simulation  results  of  a  hand  written  Chinese  character. 
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Implementation  of  Thinning  Processor  with  CNN 
Bj32  CA  uses  the  following  Boolean  function: 

^  <i)  =«/■'  •( 

where  O  is  the  Equivalence  operator.  The  schematic  of  the  above  Boolean 
function  implemented  by  logic  gates  is  shown  in  Fig.  5.  It  is  known  that 
CNN  can  finish  logical  operations  like  AND,  OR  and  NOT  [8].  Thus,  the 
above  Boolean  function  can  also  be  implemented  by  using  CNN  cloning 
templates.  Fig.  6  shows  the  whole  schematic  of  our  proposed  image 
thinning  processor.  The  X-CA  and  Y-CA  are  basically  time-discrete  CNN 
[9]. 


To  demonstrate  the  performance  of  our  binary  image  thinning 
processor.  Fig.  7  shows  the  simulation  results  of  two  hand  written  English 

characters  "a"  and  "y"  in  a  40x40  pixels  plane.  In  fact  the  size  of  a  convex 

polygon  in  the  X  and  Y  directions  can  also  be  determined  by  our  thinning 

processor.  Fig.  8  shows  some  examples  of  determining  the  sizes  of  convex 

polygons. 


By  separating  the  2-D  thinning  process  with  two  1-D  thinning  processor, 
the  whole  image  processing  is  speed  up.  In  general,  the  convergent  steps 
are  less  than  n  for  a  nxn  pixels  plane.  However,  there  are  some  minor 
disadvantages  in  our  processor:  (i)  two  layers  of  processing  elements  are 
needed,  and  (ii)  the  final  thinning  outcome  may  be  discontinueu. 
However,  the  simplicity  of  implementation  and  higher  processing  speed 
makes  it  a  vital  component  for  image  processing.  Our  research  for  the 
properties  of  B-rule  CA  and  its  application  to  image  processing  is  only  the 
first  step.  Another  example  of  applying  CA  rules  to  the  design  of  CNN  for 
image  processing  like  edge  detection  and  connected  component  detector 
can  also  be  found  in  [10],  The  relations  between  CA  and  CNN  and  their 
potential  capability  of  combining  together  for  image  processing  worth 
further  studying. 


Conclusions 

A  class  of  new  CA  (B-rule)  is  ideal  tool  for  the  design  of  CNN  for 

binary  image  processing  (like  image  thinning).  The  potential  capacity  of 

our  approach  worth  further  studying. 
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Class 

B-rules 

1 

0,  32,  72,  127,  160 

2 

4,  36,  76,  104,  108,  132,  164,  200, 
204,  232,  236 

3 

8,  22,  50,  54,  146 

4 

150,178,182,218,250,254 

5 

90,  94,  122,  126 

Table  1 


Fig.  1  Some  evolution  examples  of  rule  6532  CA 


2DCA  =>X-1DCA  +  Y-1DCA 

Fig.  2  Transformed  from  2DCA  into  X-IDCA  and  Y-IDCA 
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Fig.  8  Determining  the  sizes  of  convex  polygons  by  CNN  processor 
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1 .  Abstract 

Computer  Integrated  Manufacturing  (CIM)  Systems  having  determining  role  in  the  modem 
industry.  These  systems  contain  essentially  two  different  equipments  for  transporting  materials 
between  workstations: 

-  conveyors  can  be  used  if  the  path  of  transportation  is  fixed  and  known  in  advance,  or 

•  if  the  path  of  the  transportation  may  change  or  there  are  various  tasks  to  be  solved  along 

the  path,  the  use  of  Automatic  Guided  Vehicles  (AGV)  is  strongly  recommended. 

The  path  control  of  an  AGV  in  CIM  systems  is  generally  solved  either  by  following  an 
inductive  wire  or  by  determining  its  position  and  orientation  from  signals  provided  by  transmitters 
mounted  at  some  characteristic  points  of  the  workshop.  Both  of  these  basic  methods  have  their 
advantages  and  drawbacks. 

In  this  paper  a  new  method  will  be  proposed  for  path-control,  which  combine  the  flexibility 
and  easy  installation  of  optical  methods  with  simplicity  and  robustness  of  the  Inductive  method. 
Using  a  new  computing  paradigm,  the  Cellular  Neural  Network  (CNN)  [11,[2]  and  a  related 
device  [3],  the  VLSI  CNN  chip,  a  very  high  speed  solution  can  be  achieved,  that  is  less  expensive 
than  the  conventional  methods  -  keeping  their  advantages.  This  AGV  control  comply  with  the 
requirements  of  CIM  systems.  Further  advantages  of  the  proposed  system  are  as  follows:  fault 
tolerance  and  abflity  to  give  instructions  along  the  path,  and  the  use  of  a  simple  local  control. 

2.  The  proposed  new  method  ' 

In  this  report  the  possibility  and  feasibility  of  using  CNNs  for  control  of  AGV  is  investigated. 
The  schematics  of  the  control  unit  can  be  seen  in  Fig.l .  The  input  sensory  array  may  work  either 
in  the  visible  electromagnetic  range  or,  eventually,  in  the  ultraviolet  or  infrared  spectral  range.  In 
this  application,  the  path  determined  by  series  of  control  patterns  and  the  AGV  has  to  trace 
these,  and  deduce  the  motion  speed  and  direction.  The  input  picture  obtained  is  directly 
fonwarded  to  the  CNN  unit.  The  CNN  unit  has  to  solve  the  pre-processing  (filtering,  noise- 
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reduction,  pattern  restoration,  etc.)  of  the  detected  picture  and  the  analysis  of  the  shape,  size, 
and  orientation  of  control  patterns.  A  simple  logic  makes  the  desired  control  signal  from  the 
result  of  analysis.  This  method  is  new  in  control,  because  CNN  was  never  used  in  this  field.  The 
results  of  using  CNN  for  character  and  pattern  recognition  are  applied  combining  with  optical 
feedback. 

The  main  advantage  of  this  method: 

-  using  analog  VLSI  and  integrated  device  (see  Part  5)  a  real  time  control  available; 

-  it’s  more  flexible  and  less  expensive  than  conventional  methods; 

-  easy  installation  and  variability; 

-  fault  tolerance  due  to  robustness  of  CNN  and  optical  detection; 

-  improvements  can  be  made  easily  due  to  the  wide  application  of  CNN  (see  Part  5). 

The  control  hardware  is  less  complex  compared  to  conventional  methods. 

3.  Determination  and  analysis  of  control  pattern 

For  the  proposed  method  a  control  pattern  had  to  be  designed,  which  meets  the  folloving 
conditions: 

-  it  doesn't  contain  small  parts,  it  is  strongly  fault  tolerant  and  should  contrast  with  the 

background: 

-  the  desired  direction  can  be  determined  from  the  induvidual  pattern; 

-  it  is  symmetric  so  that  the  path  can  be  followed  there  and  back. 

Keeping  in  mind  these  conditions,  the  elementary  control  pattern  has  been  choosen  as 
depicted  in  Fig.2.,  and  its  horizontal  projection  represents  the  desired  direction.  This  pattern  is 
contiguous  therefore  it  can  be  filtered  easily.  The  horizontal  projection  can  be  calculated  by  the 
well  known  CCD  function  [9].  The  pattern  has  three  main  parameters  :  a.  />,  and  f.  They  define 
the  figure  unambiguously.  The  absolute  size  of  the  pattern  (parameter  a)  can  be  set  according  to 
the  size  of  the  visual  field  of  the  detector.  The  horizontal  projection  of  the  pattern  is  determined 
by  the  length  of  its  short  and  long  diameter  when  the  pattern  is  rotated  in  the  sensory  field.  The 
tangent  of  the  desired  path  direction  is  represented  by  the  position,  when  the  two  diameter’s 
horizontal  projections  are  equal,  so  the  AGV  has  to  turn  on  the  right  when  the  projection  is 
shorter,  and  on  the  left  when  it  is  longer. 

The  length  of  the  projection  : 

h(9)=  max^  sin(9+/3),  ysin(f +pj 

where  cf  is  the  rotation  acording  to  the  direction  to  be  followed,  and  it  can  be  in  the  intetvall 
[-«,  90*-/3]  -  otcan  be  computed  from  the  parameters  /land  jf.  Using  the  pattern  described  above 
and  it’s  horizontal  projection,  the  inverse  of  the  last  function  can  be  used  to  determine  the 
orientation  with  respect  to  the  tangent  of  the  desired  path  (Fig.3.). 
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4.  Experiments 


To  investigate  the  proposed  method  three  different  experiments  have  been  carried  out  in 

order. 


4.1  Experiment  1. 


The  goal  of  this  experiment  is  to  prove  that  the  control  pattern  can  be  detected  and 
processed  in  noisy  background,  and  verify  that  the  information  can  be  decoded  from  the 
detected  picture.  According  to  the  principles  stated  above  as  well  as  to  practical  considerations, 
the  main  parameters  of  the  control  pattern  have  been  chosen  to: 


a  =  5.25cm,  /5  =  41.88"  .  /•=57.11^ 


A  series  of  control  patterns  has  been  scanned  by  means  of  a  camera.  An  individual  control 
pattern  has  been  cut  out  from  the  stream  and  digitized  with  a  frame  grabber  (Fig.4.).  The  noisy 
image  of  resolution  44x44  has  been  fed  into  a  CNN  program  package  [8].  The  simulator 
computed  and  displayed  the  transient  of  the  CNN  at  discrete  time-intervalls.  After  the  transient 
has  settled  down,  the  steady  state  response  of  the  CNN  remains  on  the  screen.  In  reality,  using 
analog  VLSI  realization  of  CNN,  this  transient  settles  down  in  a  few  microseconds.  For  filtering 
and  CCD  function  the  following  well  known  templates  were  used  consecutively  [1  .b][9]: 


filtering; 


0  0  0 

‘.25  .25  .25 

A= 

0  2  0 

B= 

.25  2  .25 

0  0  0 

.25  .25  .25 

CCD;  A= 

b  0  o' 
12-1 

B=. 

b  0  b 
0  0  0 

0  0  0 

0  0  0 

1=2; 


1=0; 


The  simulation  results  clearly  shows  that  the  control  pattern  can  be  succesfully  filtered  and 
restored  from  a  noisy  background  (Fig.5.).  The  projection  length  computed  by  the  CNN  simulator 
and  displayed  at  the  right  side  of  the  output  picture  (Fig.6.)  is  proportional  to  the  angle  of  rotation 
needed  to  follow  the  path. 

4.2  Experiment  2. 

The  main  goal  of  this  experiment  was  to  investigate  whether  the  method  can  be  applied  for 
controlling  a  real  robot  or  not.  The  schematics  of  the  experimental  arrangement  can  be  seer,  in 
Fig.7.  The  camera  scan  a  control  pattern  representing  a  part  of  the  path  to  be  followed.  The 
picture  is  fed  into  a  digitizer  and  the  digitized  picture  is  processed  by  an  IBM-AT  compatible 
computer.  A  special  add-on-board  [12]  is  used  for  the  simulation  of  a  CNN  -  it  computes  the 
result  faster  than  the  software  simulator  applied  in  experiment  1 .  The  result  of  which  is  the  angle 
calculated  from  the  horizontal  projection  of  the  scanned  pattern  is  fed  into  the  control  of  a 
SCARA  type  robot,  and  the  gripper  of  the  robot  moving  along  an  arch  required,  and  represented 
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by  the  scanned  pattern.  If  the  camera  mounted  directly  at  the  gripper,  this  experimental 
arrangement  can  be  used  for  teaching-in  as  well. 

4.3  Experiment  3. 

A  software  simulator  program  has  been  developed  to  test  the  proposed  AQV  control.  On 
the  screen  the  simulator  shows  on  the  screen  the  patterns  indicating  the  desired  path,  the 
simulated  image  seen  by  a  camera  attached  to  the  vehicle,  yand  the  path  wandered  by  the  AGV. 
Using  this  simulator  the  effect  of  different  parameters  (e.g.  acceleration  of  the  vehicle,  fine 
tracking,  etc.)  can  be  tested  along  any  path  given. 

5.  Application,  further  improvements 

The  VLSI  and  optical  realization  of  CNNs  are  under  extensive  research  [7][10][11).  The 
advantage  of  the  presented  method  is  that  all  the  necessary  electronic  units  -  exept  for  power 
electric  parts  -  can  be  integrated  into  one  single  VLSI  chip.  The  chip  would  consists  of  two  CNN 
layers  performing  image  restoration  (filtering,  etc.)  and  pattern  recognition  and  interpretation 
(CCD).  To  facilitate  additional  functions  described  below  a  CNN  layer  with  programable 
templates  is  recommended.  Integrating  an  additional  sensory  array  in  CNNs  is  already  the 
subject  of  major  research.  Further  simple  logical  contrd  u'fts  may  be  attached  to  the  design  as 
well,  to  obtain  a  dual-computing  intelligent  sensory  array. 

One  additional  improvement  to  the  method  may  be  the  ’fine  tracking’.  In  the  proposed 
method  significant  rotational  deviations  {(f>  apx.  1*-  depending  on  resolution  of  sensory  array 
and  the  CNN)  can  be  corrected  easily.  However,  especially  along  a  long  straight  path,  the 
necessity  of  fine  tracking  may  be  demanded.  A  signal  for  fine  tracking  can  be  obtained  from  the 
analysis  of  the  bottom  line  of  the  CNN  image  after  filtering:  any  little  part  of  a  control  pattern 
detected  in  the  "n"  pixel-wide  left/right  margin  would  generate  an  appropriate  control  signal. 

Besides  the  proposed  method  and  device  with  additional  CNN  functions  -  e.g.  character 
recognition  -  can  be  applied  for  interpreting  other  patterns  placed  among  path-control  patterns. 
In  this  way  the  AGV  or  the  robot  might  recive  instructions  to  perform  various  tasks  like  "stop  for 
'n'  seconds",  "do  something",  "act  according  to  the  environment",  "determine  the  control 
strategy",  etc.  For  example,  the  control  pattern  may  be  combined  with  vertical  stripes,  that  don't 
influence  the  orientation  and  the  horizontal  projection  of  the  pattern.  The  number  of  stripes  may 
encode  different  instructions,  and  can  be  interpreted  using  vertical  CCD.  Further  investigations 
are  needed  concerning  the  fault  tolerance  and  robustness  of  such  "enhanced"  patterns. 
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Figure  l  Sematic  of  control  unit 


Figure  2  Elementary  control  pattern 


Figure  4  Digitized  noizy  input  frame 


Figure  S  The  filtered  pattern 


Figure  3  Desired  path  represented  by  control  patterns 
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/Uostract 

The  random-dot  stereogram  coding  3D  Information  In  Its  Internal  correlation  Is  for  probing 
human  stereopsis.  We  report  dual  CNN  algorlttims  that  can  reveal  3D  surfaces  coded  in 
stereograms.  The  concept  of  difference  stereogram  is  introduced  and  used  for  coding 
smooth  surfaces.  Its  importance  is  due  to  the  fact  that  difference  stereograms  of  real 
objects  can  be  created  in  an  optical  environment  using  projector  and  camera. 

INTRODUCTION 

An  interesting  question  about  the  human  visual  system  is  whether  we  identify  an  object  before 
we  put  it  in  its  proper  perspective,  or  whether  there  exists  a  mechanism  -  the  Cyclopean  eye  -  in  the 
visual  cortex,  that  can  perceive  “pure"  depth  based  merely  on  the  correlation  and  parallax  of  the  left  and 
right  retinal  images.  Using  random-dot  stereograms  [4]  almost  anyone  can  try  and  testify  to  oneself  the 
existence  of  the  Cyclopean  mechanism. 

The  random-dot  stereogram  (RDS)  is  a  two-dimensional,  seemingly  random  pattern  consisting 
of  pairwise  horizontally  correlating  internal  segments.  Shifting  a  RDS  horizontally  upon  itself,  correlating 
(identical)  areas  will  overlap  again  and  again.  Viewing  properly  an  RDS  devoid  of  any  (monoculariy) 
observable  cue,  the  Cyclopean  eye  will  find  the  correlating  areas  and  “see"  them  in  different  perspective 
depths,  depending  on  their  horizontal  distance  on  the  stereogram. 

In  Section  1,  methods  of  creating  different  types  of  RDS’s  -  stereo  pairs,  auto-stereograms  and 
difference  stereograms  -  are  discussed,  laying  stress  upon  the  internal  structure  of  stereograms. 

In  Sections  2  and  3,  dual  CNN  algorithms  that  can  find  the  perspective  hidden  in  stereograms, 
are  dealt  with. 

In  Section  4,  an  optical  set-up  with  projector  and  camera,  to  produce  difference  stereograms  of 
real  objects  is  outlined.  Potentially,  if  connected  to  a  dual  CNN  hardware  it  can  detect  3D  depth 
variations  In  its  scope. 

1.  DIFFERENT  TYPES  OF  RANDOM-DOT  STEREOGRAMS 

The  RDS  is  a  visually  perceptible  rectangular  pattern  printed  on  paper  or  displayed  on  a  screen, 
conveying  3D  depth  information  coded  in  its  internal  correlation.  Its  mathematical  representation,  which 
for  simplicity  we  also  call  RDS,  is  a  two-variable  function  g(x,y)  coding  a  depth  pattern  represented  by 
the  s(u,v)  surface  function.  In  visualization  the  g(x,y)  values  are  converted  into  different  visual  attributes 
as  colour,  brightness  or  texture. 

In  RDS’s,  depth  is  transformed  into  correlation,  which  is  defined  as  follows: 

g  at  (x^y)  and  (Xf.+d,y)  is  fully  correlated,  if  g(x,y)=g(x+d,y)  for  x^^f.+l,  where  /  is  a  suitable 
correlation  length.  (For  partial  correlation  the  equality  need  not  hold  uniformly  over  the  total  correlation 
length.) 
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On  its  either  -  say  left  -  side,  the  RDS  contains  a  rectangular  area  with  no  correlation  inside. 
Almost  all  the  rest  of  RDS  is  composed  of  patches  that  are  copies  of  some  other  areas  to  the  left  of 
them.  The  copy  mechanism  ensures  internal  correlation  in  the  above  sense.  The  position  of  areas  to  be 
copied  is  determined  in  accordance  with  the  depth  pattern  to  be  coded. 

The  different  types  of  RDSs  are  defined  by  the  following  formulae: 

Let  Xq  and  Yq  respectively  be  the  horizontal  and  vertical  dimensions  of  the  RDS,  and  r(x,y)  an  internally 
uncorrelated  pattern  for  0^<Xq  and  Q^<Yd,  and  let  Xp  be  the  width  of  the  left  uncorrelated  area. 
We  define  the  S(u.v)  partial  slope  function  of  the  surface  in  the  horizontal  direction  as  the  limit  of 
(s(u + D,v)-s(u,v))/D  with  D  approching  0.  We  also  introduce  a  c(s)  copy-source-selector  function  to  be 
defined  later,  according  to  the  different  types  of  RDSs. 

Then  for  0<y< Vo,  g(x.y)=r(x,y)  ifx<xp, 

g(x.y) = g(c(s(x-Xp,y)),y)  if  x>Xp  and  S(x-Xp,y)  <  1 , 

g(x,y)=r(x,y)  if  and  S(x-Xp.y)>1. 

All  the  quantities  above,  i.e.  coordinates,  RDS  and  depth  values  can  be  either  real  or  integer 
values.  The  width  Xp  sets  constraints  upon  the  dynamics  of  depth  variations  for  the  sake  of  a  meaningful 
resulting  stereogram. 

\fxQ=2*Xp,  we  obtain  a  random-dot  stereopair,  with  either  half  uncorrelated  in  itself.  For  visual 
inspection  its  halves  can  be  separated  and  projected  onto  the  left  and  right  retinae. 

If  Xq  is  some  multiple  of  Xp,  we  obtain  an  auto-stereogram  [5],  that  can  be  inspected  without  any 
special  device,  simply  fixating  behind  or  before  the  plane  of  the  RDS.  The  two  fixation  positions  give 
reverse  depth  perception  with  respect  to  each  other.  Fig.  2  shows  the  auto-stereogram  of  the  step- 
pyramid  seen  in  Fig.  1 . 

For  stereopairs  and  auto-stereograms  the  copy-source-selector  function  is  the  following: 
c(s)-x-Xp+s. 

Using  c^sj  =  {'x-Xp-hs^  mod  Xp  as  the  copy-source-selector  function  leads  to  difference 
stereograms.  The  difference  stereogram  is  for  coding  the  changes  in  surface  depth,  instead  of  the  depth 
itself.  It  has  two  distinct  advantages  from  the  aspect  of  depth  coding.  One  is  the  reduced  requirement 
for  width  Xp  due  to  the  fact  that  changes  of  depth  on  surfaces  are  usually  smaller  than  the  absolute 
depth.  The  other  is  that  difference  stereograms  can  be  produced  by  simple  optical  devices  too,  as  will 
be  seen  later. 


2.  DUAL  CNN  ALGORITHM  TO  EXTRACT  DEPTH  FROM  RDS’S 

When  perceiving  depth,  the  binocular  fusion  mechanism  of  the  human  visual  system  picks  out 
fields  correlating  in  the  two  retinal  images.  Correlation  is  tested  by  overlapping  the  retinal  images  at 
different  relative  shifts.  Based  on  an  entropy-like  measure  -  called  neurontropy  [6]  -  maximum 
correlating  areas  together  with  their  relative  shifts  which  are  directly  proportional  to  the  depth  of 
corresponding  areas  on  the  coded  surface,  are  extracted. 

The  function  of  this  neural  mechanism  can  be  broken  into  steps  realizable  in  the  framework  of 
the  Dual  CNN  paradigm  [2],  which  contains  logic  operations  in  addition  to  the  analog  CNN  ones.  Binary 
(say  black  on  v/hite)  random-dot  stereograms  coding  surfaces  that  are  discretized  in  ail  three 
dimensions,  can  be  processed  in  the  dual  CNN  framework. 
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The  dual  CNN  algorithm  can  find  the  surface  steps  in  depth,  one  after  the  other,  in  successive 
phases.  Each  phase  results  in  the  image  of  surface  segments,  the  fragments  of  the  whole  surface  that 
lie  in  the  depth  layer  just  investigated.  Since  in  the  dual  CNN  framework  [3]  an  external  control  is 
supposed  to  exist,  the  value  of  depth  just  investigated  is  always  known.  Combining  the  results  of 
successive  phases,  yields  also  the  level  line  structure  of  the  whole  surface. 

In  the  following  outline  of  the  algorithm,  the  RDS  is  split  into  left  and  right  images.  Both  images 
and  the  results  as  well,  are  XQ-Xp  wide,  the  left  image  being  the  left  part  of  the  RDS  and  the  right  one  its 
right  part  of  the  same  width. 

The  N-th  phase  of  the  algorithm  consists  of  the  following  steps: 

Step  1.;  We  shift  the  right  image  by  one  pixel  to  the  left  and  replace  it  with  the  result.  Depending  on  the 
dual  CNN  device  used  shifting  can  be  performed  in  two  different  ways.  If  the  right  image  can  always  be 
stored  inside  the  dual  CNN  -  e.g.  in  its  logic  register,  then  a  simple  template,  having  only  one  non-zero 
entry,  833=  1.  will  do  the  shift.  However,  if  the  right  image  is  stored  in  external  memory,  loading  it  into 
the  CNN  from  gradually  increasing  start  addresses  will  substitute  for  the  shift. 

Step  2.:  The  task  to  find  in  the  left  and  right  image,  sufficiently  large  correlating  areas  having  the  same 
pattern,  involves  the  following  steps. 

Step  2/a.:  We  perform  a  logical  equivalence  operation  that  by  making  pixel  level  correlations,  will 
immediately  reveal  to  an  onlooker,  the  areas  being  searched  for.  The  image  obtained  is 
contiguously  black  on  correlating  areas  and  shows  random  black  and  white  noise  pattern 
elsewhere. 

Step  2/b.:  To  demarcate  correlating  areas  from  noise  by  CNN,  we  introduce  the  next  simple  criterion: 
the  black  pixels  having  no  white  neighbours  (most  probably)  belong  to  correlating  areas.  Using 
the  edge  [7]  template,  that  inverts  the  internal  pixels  in  black  areas,  we  can  find  just  those 
pixels  that  fulfil  the  criterion. 

Step  2/c.:  The  final  step  in  finding  correlating  areas  is  the  removal  of  the  noisy  pattern.  Combining  by 
logical  or  operation,  the  result  of  edge  (Step  2/b)  and  the  inverted  result  of  logical  equivalence 
(Step  2/a)  will  yield  the  white  image  of  the  N-th  layer  of  the  coded  surface,  on  a  black 
background. 

The  logical  inversion  and  logical  or  operations  can  be  performed  consecutively  by  the  logic 
part  of  dual  CNN.  Equivalently,  a  composite  inverse-or  operation  can  be  defined  and  performed 
using  the  (A22=  1,  ^ 22^-1.  /=  V  analog  template. 

The  success  of  this  noise  removal  step  is  highly  dependent  on  the  local  randomness  of  RDS,  or 
-  conversely  -  on  the  adequacy  of  the  demarcation  criterion  above.  According  to  experience, 
small  error  patches  may  remain  in  the  results.  Appiving  a  small  positive  value  in  the  feedback 
off-center  positions  of  the  inverse-or  temf^ate  can  improve  noise  removal  by  filtering  out 
isolated  errors  or  even  one  pixel  wide  stubs,  depending  on  the  value  applied. 

Step  3.:  Starting  in  the  first  phase  with  an  ‘empty  black  image,  the  surface  layers  found  in  succession 
can  be  aggregated  by  logical  and  operation  to  produce  the  level  line  structure  of  the  surface  coded. 
Essentially,  the  succession  of  logical  ands  cuts  and  pastes  those  edges  that  were  fourKf  in  the 
demarcation  step  (2/b).  In  addition,  it  has  a  noise  filtering  side  effect  in  the  sense  that  any  errors  that  lie 
entirely  inside  areas  on  other  depth  iayers  (before  or  behind  in  depth)  will  disappear.  On  the  other  hand, 
errors  that  cross  surface  edges  wili  iocally  corrupt  the  ievei  iine  resuit.  Fig.  3  shows  the  ievel  lines  of  the 
step-pyramid  seen  in  Fig.  1 . 
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3.  THE  DIFFERENCE  RDS  AND  ITS  USE  WITH  REGULAR  DOT  PATTERN 


Producing  a  difference  RDS  effectively  results  in  the  storage  of  the  difference  betw/een  the  depth 
of  each  fragment  and  another  fragment,  a  distance  Xp  to  the  left.  In  other  words,  the  difference  RDS  of  a 
surface  is  the  auto-stereogram  of  the  "difference"  surface,  that  can  be  produced  simply  if  we  divide  the 
original  surface  into  xp  wide  vertical  bands  and  -  keeping  the  leftmost  band  unchanged  -  reduce  the 
depth  of  each  fragment  by  the  depth  of  the  corresponding  fragment  in  the  adjacent  band  to  the  left. 
Using  a  small  enough  Xp  with  respect  to  the  minimum  horizontal  distance  of  unit  depth  changes,  yields 
a  three-level  (-1,0,  +  1)  difference  surface.  The  surface  of  only  2  bit  dynamics  can  be  coded  with  small  Xp 
and  vice  versa,  that  allows  fine  spatial  resolution.  Fig.  4-a  and  Fig.  4-b  show  the  difference  surfaces  of 
the  76  pixel  wide  step-pyramid,  forxp=2  and  Xp  =  12,  respectively. 

From  the  aspect  of  processing  RDS’s  with  a  CNN,  the  difference  RDS,  due  to  its  narrow 
correlation  band,  is  fundamentally  different  from  the  other  types  of  RDS’s.  The  global  correlation 
property  of  those  is  transformed  into  a  local  one  of  difference  RDS’s,  that  can  be  detected  in  a  simple 
CNN  window  of  neighbourhood  2  or  3.  By  that  way,  both  flat  areas  and  ascending  or  descending 
slopes  of  the  coded  surface,  can  be  extracted  in  respective,  single  CNN  steps  directly  from  the  RDS. 

In  case  of  narrow  band  difference  RDSs  the  randomness  of  the  pattern  looses  its  importance. 
Just  the  knowledge  of  pattern  structure  may  contribute  to  the  reconstruction  of  the  coded  surface.  Let 
us  consider  a  discretized  surface  that  is  "smooth”  with  respect  to  the  pattern  density.  (Smoothness 
means,  that  the  depth  difference  of  neighbouring  pixels  are  at  most  1,  and  none  of  the  2  by  2  pixel  areas 
form  a  saddle.)  Let  us  produce  a  narrow  band  difference  stereogram  oi  Xp= 4,  using  a  regular  checker 
pattern  of  squares  of  2  pixel  long  edges  (Fig.  5).  Smooth  surfaces  coded  in  checker  pattern  can  be  fully 
reconstructed  by  a  dual  CNN  algorithm  with  22  analog  templates  of  neighbourhood  1,  in  3*3  window, 
as  follows: 

Step  1:  Find  vertical  edges  where  the  surface  is  ascending  from  left  to  right 
Step  2:  Find  vertical  edges  where  the  surface  is  descending  from  left  to  right 
Step  3:  Find  all  the  horizontal  edges 

Step  4:  Using  the  "smooth  surface"  property  and  the  already  known  sort  of  vertical  edges,  identify  the 
downward  descending  and  downward  ascending  horizontal  edges,  respectively. 

Result  1 :  The  union  of  edges  found  gives  the  level  lines  of  the  surface  (Fig.  6) 

Step  5:  Propagating  the  shadow  of  ascending/descending  edges  until  it  reaches  a 

descending/ascending  edge  gives  the  ascending/descending  slopes  both  in  horizontal  and 
vertical  direction. 

Step  6:  The  intersection  of  ascending  and  descending  slopes  gives  the  row-wise/column-wise  local 
maxima/minima,  in  other  words  the  vertical/horizontal  ridges/valleys  of  the  surface. 

Step  7;  Calculate  the  intersection  of  vertical  and  horizontal  ridges/valleys.  Erase  those  parts  of  the 
result  which  are  not  bordered  by  some  closed  level  line  loop  exactly. 

Result  2:  Result  of  Step  7  yields  the  local  maxima/minima  (peaks/hollows)  of  the  surface. 

Result  3;  Using  modified  CCD  to  propagate  ascending/descending  edges  until  reaching  a 

descending/ascending  edge,  the  height/depth  of  local  maxima/minima  can  be  measured,  from 
the  direction  of  propagation  (Fig.  7). 
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4.  THE  CREATION  OF  DIFFERENCE  RDS  WITH  OPTICAL  MEANS 


Aside  from  its  inevitable  importance  in  vision  research,  up  until  this  point  to  produce  and  probe 
RDS  may  have  seemed  an  enjoyable  and  aesthetic  play,  devoid  of  any  significance  from  the 
engineering  point  of  view.  However,  in  principle,  diffeience  RDS’s  of  real  objects  can  be  produced 
directly  by  optical  means.  Moreover,  research  of  optical  CNN  devices  is  reportedly  under  way  [8],  that 
together  with  the  possibility  of  making  continuous,  real  time  difference  RDSs  of  surrounding  objects 
may  lead  to  a  "stereoscopic"  robot  eye. 

Let  us  consider  a  projector  and  a  camera  side  by  side  at  the  same  height  in  front  of  a  white  wall. 
Let  us  project  perpendicularly  onto  the  wall  a  horizontally  periodic  random-dot  (or  checker)  pattern,  i.e. 
the  RDS  of  a  single  plane,  from  such  a  distance  that  the  parallax  over  the  projection  area  be  negligible. 
The  camera  -  from  beside  the  projector  -  will  see  the  whole  area  at  a  spec'ific  angle,  that  depends  on  its 
distance  from  the  projector,  and  is  the  same  for  the  whole  area.  In  fact,  the  camera  will  see  the  RDS,  or 
what  is  the  same  in  this  case,  the  difference  RDS  of  a  single  plane. 

Let  us  place  some  angular  objects  in  front  of  the  wall  in  such  a  way,  that  their  surfaces  be  either 
parallel  with  or  perpendicular  to  the  wall.  From  the  projectc.  no  change  in  the  projected  pattern  can  be 
noticed.  However,  from  the  camera,  definite  horizontal  shifts  of  pattern  segments  that  now  fall  on  the 
inserted  objects,  can  be  detected.  The  resulting  view  from  the  camera  is  the  difference  RDS  of  the 
whole  arrangement,  except  for  the  images  of  those  surfaces,  which  are  perpendicular  to  the  wall.  They 
go  "empty"  in  the  optical  method  and  would  be  filled  with  an  uncorrelated  pattern  in  the  computer 
algorithm.  A  camera  image  of  objects  in  Fig.  8-a  is  shown  in  Fig.  8-b.  The  amount  of  horizontal  shift  is 
determined  by  the  change  in  depth  and  the  camera  angle.  We  consider  depth  resolution  to  be  that 
change  in  depth,  which  causes  one  pixel  horizontal  shift  in  the  camera  image.  For  a  given  pixel  size,  the 
depth  resolution  can  be  adjusted  by  the  camera  angle. 

Presumably,  using  electronically  controlled  devices,  the  surface  in  front  of  the  above  set-up.  can 
be  scanned  row-wise  by  a  flying  window  and  processed  in  real  time  by  a  tiny  CNN  array.  Moreover,  in  a 
feedback  loop,  the  resolution  and  the  size  of  the  flying  window  can  be  adjusted  adaptively  to  the 
extracted  local  features  of  the  surface. 
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Abstract 

This  paper  describes  a  flexible  test  set  that  allows  PC-controlled  analog  and  digit2d  mea¬ 
surements  of  a  test  object.  The  concept  is  modularly  structured  and  enables  an  individual 
constellation  depending  on  the  testing  object  such  as  a  CNN  chip.  Analog  and  digital  signals 
can  be  transferred  to  and  from  the  object.  The  communication  to  the  PC  is  realized  via  a 
standard  interface.  For  the  PC,  a  test  program  language  (TPL)  has  been  developed,  which 
is  easy  to  handle  and  allows  a  flexible  and  completely  automatic  testing  of  a  chip.  The  re¬ 
sults  of  the  measurement  (e.g.,  dc  characteristics)  can  be  graphically  visualized.  For  dynamic 
measurements  the  bottle  neck  consists  in  the  data  transfer  to  the  PC  and  the  processing  of 
the  testprogram  interpreter.  To  overcome  this  problem  special  hardware  modules  have  been 
developed,  which  allow  a  cyclic  or  linear  output  of  analog  and  digital  signals  up  to  10  MHz. 
Their  core  consists  of  a  FIFO,  which  is  preloaded  before  the  processing.  The  output  rate  is 
controlled  by  a  clock  and  different  modules  sure  synchronized  via  trigger  signals. 


1  Introduction 

As  commercial  chip  testers  are  very  expensive,  the  necesaty  to  develop  a  economic  test  set  for 
the  test  of  the  first  DTCNN  chip  [1]  was  ©ven.  The  idea  was  to  built  up  a  system  that  is  not 
fixed  to  a  special  structure  of  the  test  object  [2].  It  was  designed  with  respect  to 

•  a  simple  measurement  by  PC  with  a  minimal  number  of  additional  wire  connections 

•  a  flexible  and  expandable  structure  in  hardware  and  software 

•  an  automatic  DC  characteristic  measurement. 

•  a  dynamic  testing  up  to  lOMHz. 

•  a  software  with  an  interface  to  C-routines. 

This  has  been  achieved  by  a  modular  structure,  where  the  test  set  consists  of  a  number  of  single 
modules,  which  communicate  with  the  PC  via  a  standard  periphery  bus  by  the  I/O  card  [3],  [4]. 
Each  module  has  an  identification  address  and  a  status  re^stcr.  Thus  the  operation  of  multiple 
modules  of  the  same  type  is  possible.  They  are  connected  by  a  port  or  a  BNC  connector  to  the 
test  object  (e.g.,  a  CNN  chip),  which  is  plugged  on  a  connection  card.  The  advantage  of  this 
concept  is  that  only  the  connection  card  has  to  be  rebuilt  for  a  new  chip.  The  required  modules 
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are  simply  plugged  on  the  ports  of  a  bus  card.  For  additional  requirements,  new  modules  can 
be  developed,  if  the  same  bus  protocol  is  used. 

Closely  related  to  the  flexible  hardware  structure,  the  software  has  to  support  this  concept  in 
flexible  and  expandable  routines.  It  is  based  on  the  assignment  of  logical  signal  names  to  hard¬ 
ware  addresses.  A  hardware  configuration  list  allows  the  user  to  operate  ordy  with  logcal  signal 
names.  Each  signal  is  uniquely  assigned  to  a  channel  on  a  specified  module  So,  a  configuration 
table  has  to  be  created  at  first  for  each  test  object,  which  also  fixes  the  required  types  and 
numbers  of  modules. 

2  Hardware  Structure 

2.1  Hardware  Cosicept 


Figure  1;  Hardware  concept  of  the  PC-controlled  testing  device. 

The  measurement  system  shown  in  Fig.  1  is  based  on  a  PC  with  a  standard  I/O  interface,  which 
provides  16  bit  output  data  forming  the  address  bus,  16  bit  bidirectional  data  realizing  the  data 
bus,  control  lines  used  for  the  handshake  protocol  and  three  timers.  An  interface  adapter  links 
the  I/O  port  to  the  system  bus.  It  buffers  the  data  and  control  bits,  supervises  the  timing  for 
handshaking  and  renders  the  system  clock. 

The  system  bus  is  the  common  interface  to  all  modules.  Its  timing  is  defined  by  a  special  bus 
protocol  using  a  handshake  procedure.  In  addition  to  the  I/O  bus,  it  provides  the  system  clock  of 
4MHz,  8  common  clock  lines,  8  trigger  lines  and  power  supplies.  The  external  clocks  are  used  for 
controlling  the  measurement  of  dynamic  events.  The  modules  can  be  synchronized  with  respect 
to  rising  or  falling  edges  of  a  clock  and  are  activated  by  a  trigger  impulse  on  a  trigger  line.  This 
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guarantees  correct  timing  and  interaction  of  multiple  modules.  Depending  on  the  given  task  the 
types  are  chosen  and  plugged  into  the  slots  of  the  system  bus.  The  modules  can  be  categorized 
as  follows: 

•  static  modules  writing  (reading)  digital  and  analog  data  to  (from)  the  test  object. 

•  dynamic  modules  writing  (reading)  distal  or  amalog  data  to  (from)  the  test  object. 

•  control  modules  controlling  the  system  and  the  synchronisation  of  dynamic  functions. 

The  system  is  able  to  address  32  different  modules  and  64  cards  of  the  same  type.  The  test 
object,  which  was  here  the  fabricated  DTCNN  chip  [5],  is  placed  on  an  own  routing  board  and 
connected  to  the  inputs  and  outputs  of  the  device. 

2.2  Realisation  of  the  Hardware 

The  different  modules  are  now  described  in  detail: 

Static  Functions 

Digital  Out:  This  module  writes  16  binary  outputs  to  the  test  object.  They  are  transferred 
from  the  PC  and  stored  in  a  register. 

Analog  Out:  This  module  writes  2  analog  output  channels  within  a  range  between  OV  and  5V 
to  the  test  object.  They  are  transferred  binary  with  8  bits  for  each  channel  from  the  PC 
and  are  stored  in  a  register.  Digital  data  are  transformed  into  analog  ones  by  two  8  bit 
D/A  converters  having  a  resolution  of  18  mV. 

Digital  In:  This  module  reads  16  binary  signals  from  the  test  object,  which  are  transferred  to 
the  PC. 

Analog  In:  This  module  has  16  analog  input  channels,  which  can  be  read  between  OV  and  5V. 
Two  of  them  are  selected  via  8-channel  multiplexers  and.  The  analog  signals  are  applied 
to  8  bit  A/D  converters  and  transferred  to  the  PC. 

Dynamic  Functions 

Digital  Out  Stack:  This  module  writes  16  binary  outputs  to  the  test  object,  which  are  loaded 
by  the  PC.  In  contrast  to  the  Digital  Out  module,  the  values  are  prdiminary  stored  in 
a  stack  (2048  x  16  bit).  Synchronized  to  a  trigger  signal,  the  module  starts  to  issue  the 
data  with  a  rate  up  to  10  MHz.  The  trigger  signal  is  selected  out  of  the  8  trigger  lines,  the 
output  rate  is  chosen  out  of  the  8  common  clock  lines.  Three  different  operating  modes, 
namely  the  single,  cyclic,  and  static  mode,  do  exist.  The  data  block  with  up  to  2048  values 
can  be  issued  once  or  cyclic,  where  the  stack  is  read  out  periodically.  In  the  static  mode  it 
works  like  the  Digital  Out  module. 

Analog  Out  Stack:  This  module  writes  2  analog  output  channels  within  a  range  between  OV 
and  5V^  to  the  test  object,  which  have  been  written  by  the  PC  into  a  stack  (2048  x  16  bit). 
It  works  in  the  same  way  like  the  Digital  Out  Stack,  only  the  binary  data  are  converted 
by  two  fast  8  bit  D/A  converters. 
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Digital  In  o...ck:  This  module  starts  synchronized  with  a  trigger  signal  reading  16  binary 
inputs  from  the  test  object  with  a  speed  rate  up  to  10  MHz.  The  trigger  signal  can  be 
selected  out  of  the  8  trigger  lines,  the  processing  speed  can  be  choosen  out  of  the  8  common 
clock  lines.  The  input  data  are  stored  in  a  stack  (2048  x  16  bit)  and  are  transferred  to  the 
PC  after  the  reading  operation.  The  status  empty,  half  full  and  full  can  be  checked  by  the 
PC.  If  the  stack  is  full  or  if  the  first  value  is  read,  further  input  data  will  be  ignored. 

Clock  Module:  This  module  selects  two  of  the  8  common  clock  lines  and  issues  a  dual  phase 
clock  to  the  test  object.  The  output  clock  can  be  switched  off  and  on.  It  also  allows  an 
activation  synchronized  with  the  selected  trigger  signal  that  begins  with  the  low  or  the 
high  edge  of  the  chosen  clock. 

Control  Functions 

Trigger  Module:  This  module  issues  the  trigger  signal  (250  ns  low  impulse)  for  synchroniza¬ 
tion.  It  is  provided  after  a  low  or  a  high  edge,  respectively,  on  the  specified  clock  line.  The 
reaction  can  be  delayed  up  to  128  /rs  in  steps  of  250  ns. 

Reset  Module:  This  module  generates  a  system  reset  to  force  a  defined  state  of  all  modules 
connected  to  the  bus  and  to  set  all  outputs  to  OV.  This  is  obtained  by  a  power  on  reset,  a 
software  reset  or  a  reset  butto  i.  It  also  provides  the  inputs  for  the  common  clock  lines. 


3  Software  Structure 

To  support  the  hardware  a  special  test  program  language  TPL  has  been  developed.  Its  concept 
is  illustrated  in  Fig.  2. 

It  can  be  considered  as  an  interpreter,  which  processes  the  statements  of  the  test  program  step  by 
step.  Before  the  interpreter  becomes  active,  a  compiler  checks  for  syntax  errors  and  substitutes 
the  signal  names  by  the  real  hardware  addresses.  Every  TPL  program  consists  of  two  parts.  The 
first  one  defines  the  connected  modules  and  ^ves  the  reference  between  the  logic  signal  names 
of  the  test  object  and  the  assigned  hardware  channels  of  the  testing  device  in  a  cross  table.  The 
second  part  contains  the  executing  statements  in  a  Pascal-like  notation. 

The  system  enables  interaction  with  the  user  via  the  keyboard  or  available  data  files.  The  output 
is  written  on  the  screen.  Also  graphics  are  possible  enabling  a  clearly  arranged  presentation  of  the 
obtained  results.  The  data  can  be  stored  in  files,  which  allows  postprocessing  with  commercial 
software  packages  such  as  MATLAB  or  MATHEMATICA. 

Below,  some  examples  of  statements  are  gjven  to  demonstrate  the  features  of  the  test  set.  The 
statements  are  grouped  after  their  functionality. 

I/O-statements;  PUT  signals;  GET  signals;  FPUT  signals  ;  FGET  signals;  CLEAROUT  sig¬ 
nals; 

While  the  statements  PUT/GET  write  and  read  signals  to  the  modules  Digital  In/Out  and 
Analog  In/Out,  FPUT  and  FGET  enable  to  write/  read  a  list  of  data  in  a  stack  for  dynamic 
testing.  CLEAROUT  resets  the  stack. 

Timing  and  trigger  statements:  TRIGGER  module,  clock,  number,  edge,  delay; 
STARTOUT  signals,  clock,  trigger,  mode;  STOPOUT  signals; 
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Figure  2:  Software  concept  of  the  PC-controlled  testing  device. 


TRIGGER  generates  a  trigger  signal  on  the  bus  line  number  by  a  specified  module.  The  signal 
is  synchronized  either  with  the  rising  or  falling  edge  of  a  specified  clock  and  can  be  delayed. 
STARTOUT  activates  a  stack  operation,  which  is  started  by  a  specified  trigger  pulse,  for  dynamic 
signals  to  a  given  clock.  The  mode  enables  starting,  stopping  or  a  synchronized  operation  with 
the  rising  or  falling  edge  of  the  clock.  STOPOUT  stops  the  stack  operation  of  dynamic  signals 
without  a  reset  of  the  stack. 

Programming  statements:  PROCEDURE  BEGIN  END,  FOR  DO  DONE,  DO  UNTIL 

DONE,  IF  THEN  ELSE  ENDIF 

The  statements  correspond  to  the  usual  Pascal  syntax  as  do  the  arithmetic  operators. 

File  statements:  RESET  file;  REWRITE  file;  APPEND  file;  READ  file,  signal;  WRITE  file, 
text,  signal;  CLOSE  file; 

The  TPL  enables  access  to  so-called  protocd  files.  Such  a  file  starts  with  the  name  PROT  and 
has  as  an  extension  a  number  between  0  and  999.  A  PROT  file  is  opened  for  reading  by  the 
statement  RESET  or  for  writing  by  REWRITE,  respectively.  APPEND  allows  the  appending  of 
data.  Data  are  handled  by  the  READ  and  WRITE  statement,  where  file  =  —  1  enables  reading 
from  the  keyboard  or  writing  on  the  screen. 

Graphic  statements:  GRON  xmin,  xmax,  ymin,  ymax;  GROFF;  GRPLOT  x,y;  GRLINETO 

x,y; 

GRON  switches  the  graphic  mode  on  and  defines  the  scaling.  GRPLOT  draws  a  point  x,  y  and 
GRLINETO  a  line  to  a  new  point  x,y  from  the  last  point  defined  by  GRPLOT  or  GRLINETO. 

The  language  is  expandable  for  new  modules  and  any  C-routines  can  be  embedded.  The  state¬ 
ments  are  also  available  in  a  C-library,  which  allows  the  inclusion  in  any  C-routines. 
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4  Conclusion 


An  analog  and  digital  test  set  ha^  been  introduced  providing  a  very  comfortable  and  automatic 
testing  of  CNN  chips.  The  hardware  is  structured  into  single  modules,  which  communicate  via  a 
common  system  bus  to  a  host  computer.  Depending  on  the  test  object,  the  system  configuration 
can  be  chosen  just  by  putting  the  requested  cards  into  the  slots  of  the  system  bus.  The  applied 
modules  must  be  specified  in  a  definition  part,  where  also  the  reference  between  signal  names 
and  their  assigned  hardware  channels  must  be  defined.  A  test  program  language  based  on  an 
interpreter  has  been  developed  that  uses  a  notation  similar  to  Pascal.  Thus  a  variety  of  features 
has  been  acliieved  with  only  a  small  set  of  statements,  which  is  easy  to  use. 
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Abstract 

This  paper  discusses  neural  networks  with  the  simple  piecewise  linear  limiter  (2), 
which  owes  much  of  its  popularity  to  its  application  in  the  CNN-model.  We  present  a 
new  approach  to  understanding  the  dynamical  behaviour  of  Hopfield  type  and  related 
models  with  these  limiters.  The  approach  allows  for  a  better  intuitive  understanding 
of  the  model  and  a  good  assessment  of  the  possibilities  of  the  model. 

1  A  geometrical  interpretation 

Consider  a  time-continuous  Hopfield  type  neural  network  with  piecewise  linear  lim¬ 
iters  as  described  by 

y  =  f{x) 

where 

x{t)  €  is  the  state  vector  at  time  t 
y{t)  €  R^  is  the  output  vector  at  time  t. 

N  is  the  number  of  neurons. 

L  :  R^  -+  R^  :  y  — »  Ay  -H  6  is  an  affine  operation,  specified  by  the  weight  matrix 

A  €  R^^^  and  the  vector  b  €  R^. 

f'.R^^R^: 

fi{x)  =  -1  if  Xi  <  -1 

fi{x)  =  if  -1  <  li  <  1  (2) 

fi{x)  =  1  if  li  >  1 

/  is  the  projection  on  the  closed  hypcrcube  C  =  [-1, 1]^.  A  neuron  is  said  to  be  in 
the  linear  region  if  —  1  <  Xj  <  1  and  it  is  saturated  if  >  1  or  Xj  <  —  1.  The  neurons 


=  -X  -b  i(y) 
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Figure  1:  the  piecewise  linear  limiter  fi 


on  the  boundary,  |xi|  =  1,  are  assigned  to  the  region  they  are  entering,  i.e.  to  the 
linear  region  if  x,(— a;*  -f  Li{x))  <  0,  to  the  saturation  region  if  Xi{-Xi  +  Li{x))  >  0 
and  to  both  if  Xi(—Xi  +  Li(x))  =  0. 

The  CNN  model,  as  introduced  in  [1],  is  a  special  case  of  this  model  with  struc¬ 
tural  restrictions  imposed  on  the  matrix  A.  The  input  and  the  independent  current 
sources  of  the  model  are  both  absorbed  in  the  vector  6. 


Any  system 


p=-p  +  S(p) 


where  p  €  is  called  the  state  of  the  system,  can  be  interpreted  geometrically  in 
state  space  as  “p  follows  5(p)”:  The  velocity  p  of  p  is  the  vector  which  points  to 
S(p)  when  based  at  p.  Hence  p  instaneously  evolves  towards  S(p)  with  a  velocity 
proportional  to  the  distance.  Of  course  S(p)  changes  accordingly,  such  that  p  evolves 
like  someone  following  his  shadow  S(p),  for  ever  or  until  p  and  S(p)  coincide. 


This  point  of  view  is  especially  fruitful  for  the  model  (1),  with  a  piecewise  linear 
limiter  (2)  where  the  shadow  5(x)  =  L(y)  =  L(f(x))  has  a  simple  geometrical 
interpretation,  leading  to  intuitive  insight  in  the  dynamical  behaviour.  The  output 
y  =  f{x)  is  the  projection  of  the  state  x  onto  C.  The  shadow  ^(x)  =  L{y)  is  the 
corresponding  point  (under  the  affine  transformation  L)  of  the  parallellepiped  L{C). 


Fig.  2  shows  an  easy  example  for  a  network  with  two  neurons.  A  state,  initially 
located  in  q,  starts  moving  towards  its  shadow.  As  the  corresponding  output,  moves 
along  an  edge  of  C,  the  shadow  moves  along  an  edge  of  L{C).  After  the  output  has 
reached  the  corner  p,  the  state  converges  towards  a  stationary  shadow. 


2  Benefits  of  the  approach 

A  classical  approach  to  understand  the  behaviour  of  the  model  with  piecewise  lin¬ 
ear  limiters,  which  has  been  used  in  CNN  literature  [2],  considers  the  system  as  a 
piecewise  linear  system  with  3^  different  regions,  defined  by  fixing  for  each  neuron 
whether  it  is  in  the  linear  region,  saturated  at  1  or  saturated  at  —1.  For  a  region 
with  k  neurons  in  the  linear  region,  the  output  y  lies  on  a  ^-dimensional  face  of 
C.  In  each  region  the  system  behaves  as  a  (local)  linear  system,  with  generically 
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Figure  2:  a)  the  shadow  5(x)  of  a  state  x,  b)  a  state  follows  its 


shadow 


as  many  eigenvalues  equal  to  —1  as  saturated  neurons  (the  other  eigenvalues  can 
also  equal  —1  by  coincidence).  There  is  a  one-to-one  correspondence  between  the 
3^  regions  and  the  3^  subcubes  (“hyperfaces”)  of  the  hypercube  C . 

We  can  easily  recover  this  result  by  means  of  the  geometrical  interpretation,  and 
identify  the  subspace  corresponding  to  the  eigenvalue  —1  and  the  subspace  corre¬ 
sponding  to  the  other  eigenvalues.  The  eeisiest  case  occurs  when  all  neurons  are 
saturated.  All  eigenvalues  of  the  local  linear  system  are  —1,  implying  movement 
in  a  straight  line.  In  terms  of  the  geometrical  interpretation,  the  state  moves  to¬ 
wards  a  stationary  shadow  until  either  it  finds  it  or  it  enters  another  region  (the 
shadow  starts  moving).  If  k  neurons  are  in  the  linear  region,  the  output  lies  on  a 
A;-dimensional  face  of  the  hypercube  (7,  and  the  shadow  lies  on  a  ^-dimensional  face 
of  the  paralleUepiped  L{C).  If  a  state  lies  in  the  fe-dimensional  hyperplane  contain¬ 
ing  this  face  of  L{C),  it  stays  in  it.  If  the  state  doesn’t  lie  in  that  hyperplane,  it  is 
attracted  towards  it.  The  hyperplane  is  parallel  to  the  eigenspaces  corresponding  to 
the  N  —  k  eigenvalues  (genetically)  different  from  —1.  The  eigenspace  corresponding 
to  —1  is  orthogonal  to  the  face  of  the  hypercube  C. 

The  shadow  approach  adds  global  insight  in  the  dynamical  behaviour  to  the  frag¬ 
mented  insight  of  the  piecewise  linear  approach.  To  understand  the  transition  in 
fig.  2b  of  a  neuron  from  —1  to  1  through  the  linear  region,  the  piecewise  linear  ap¬ 
proach  gives  different  explanations  depending  on  the  eigenvalues  of  the  local  linear 
system  (or  on  the  horizontal  distance  between  the  shadows).  If  the  system  has  a 
positive  eigenvalue,  the  transition  of  the  neuron  will  be  explained  as  an  exponen¬ 
tial  evolution  away  from  an  unstable  equilibrium  for  which  the  neuron  would  take  a 
vjilue  less  than  —1.  With  the  negative  eigenvalue,  there  would  be  attraction  towards 
a  stable  equilibrium  larger  than  1.  With  the  shadow  approach,  one  can  easily  see 
that  the  shadow  starts  at  the  right  hand  side  of  the  initial  state  q,  and  will  stay 
ahead  during  the  transition  to  p. 

The  boundedness  of  the  states,  as  first  proved  in  [1]  follows  easily  from  the  fact  that 
the  shadow  always  lies  in  L{C). 
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Another  merit  of  the  shadovf  approach  lies  in  the  comparison  of  the  model  (1)  with  a 
related  Brain-State-in-a-Box  model,  which  h<is  also  been  studied  in  CNN  literature 


ii  =  —u  +  f{v) 
V  =  L(u) 


(3) 


where  - 


It  is  well  known  that  the  associated  state  v  =  L(u),  behaves  exactly  according 
to  the  equation  of  the  Hopfield  model  (1).  However,  although  this  explains  some 
of  the  relations  between  both  models,  it  is  still  surprising  that  both  models  show 
similar  binary  input  output  behaviour,  when  y  is  considered  for  (1)  and  u  for  (3). 
This  fact  can  be  explained  by  observing  that  u  follows  a  shadow  f{v)  =  /(X(u)), 
which  evolves  exactly  the  same  way  as  y  in  the  Hopfield  model.  Therefore,  it  is  not 
only  possible  to  map  every  trajectory  of  u  in  the  BSB  model  onto  a  trajectory  of 
y  in  the  Hopfield  model  by  applying  f  o  L  but  in  addition  we  know  that  a  point 
of  the  former  trajectory  is  moving  towards  the  corresponding  point  of  the  latter 
trajectory.  It  follows  immediately  that  both  models  have  the  same  equilibria.  The 
attraction  basins  are  generedly  different,  although  also  this  difference  is  often  smaller 
than  expected,  since  the  location  of  the  attraction  basins  is  closely  related  with  the 
location  of  unstable  equilibria,  which  are  equal  for  both  models. 

The  main  result  thus  far  obtedned  with  the  shadow  approach  is  a  set  of  linear 
ineqtialities  guaranteeing  the  attraction  of  all  neighbouring  patterns  (differing  in 
one  neuron)  to  a  given  pattern.  A  heuristical  solution  of  these  inequalities  leads  to 
a  new  design  rule.  This  will  not  be  treated  here  since  there  is  no  direct  connection 
with  CNN.  The  results  can  be  found  in  [4]  and  a  more  detailed  account  is  given  in 
[5]  and  [6]. 


3  A  tool  for  CNN  template  analysis 

The  shadow  approach  can  also  be  a  tool  in  the  analysis  of  templates  for  CNN. 
Because  of  the  cellular  nature,  the  behaviour  of  these  models  can  often  be  understood 
by  isolating  a  small  part.  Consider,  for  example,  the  opposite-sign  templates  as 
studied  in  [2]  [7]  and  [8].  The  matrix  A  in  (1)  is  tridiagonal  with  a  constant  value  p 
on  the  diagonal,  —s  on  the  upper-diagonal  and  s  on  the  lower  diagonal  and  6  =  0. 
When  (p  — 1)/2  <  s  <p— 1,  this  system  behaves  as  a  connected  component  detector 
(CCD).  If  Xi  =  1,  it  converges  to  an  equilibrium  with  leading  neurons  1,  followed 
by  alternating  neurons  1  and  —1,  with  as  many  times  —1  as  connected  regions  of 
— 1  in  the  initial  image.  If  xi  =  —1,  it  detects  connected  regions  of  1  in  a  similar 
way.  Fig.  3a  shows  the  trivial  case  with  two  neurons  only,  where  every  pattern  is 
stable.  For  instance,  the  pattern  [—1  —  1]  should  be  read  as  “no  components  of 

ones”.  Now  we  add  a  neuron  in  front  of  the  first  two.  A  crucial  observation  for 
the  CCD  is  that  the  first  neuron  never  changes,  once  it  is  saturated.  Therefore  the 
effect  of  adding  the  first  neuron  can  be  described  as  adding  a  vector  6  to  the  system 
of  the  remaining  neurons.  Therefore,  if  a  —1  is  added  (indicating  ones  are  counted). 
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the  parallellogram  of  fig  3b,  showing  the  state  space  for  neurons  2  and  3,  is  located 
more  to  the  left,  such  that  the  pattern  [(—1)  1  1]  is  now  unstable  and  is  attracted 
towards  [(—1)  —  1  1]  to  be  read  as  “one  component  of  ones”.  If  a  1  is  added  in  front, 
the  paralleilogram  is  shifted  down  emd  the  system  similarly  counts  components  of 
— 1.  If  an  intermediate  value,  close  enough  to  0  is  added  in  front,  neurons  2  and  3 
temporarily  behave  again  as  in  the  case  of  only  two  neurons,  until  the  first  neuron 
gets  saturated  at  1  or  —1.  One  can  extend  this  argument  to  see  that  the  system 
behaves  as  a  connected  component  detector  for  any  number  of  neurons.  Finally, 
note  that  the  description  of  the  parallellogram  shifting  up  and  down,  according  to 
the  value  of  a  third  neuron  is  nothing  else  than  a  two-dimensional  representation  of 
the  parallellepiped  L{C)  in  the  threedimensional  state  space. 


4  Conclusion 

A  new  approach  to  understanding  the  dynamical  behaviour  of  Hopfield  type  and 
BSB  type  neural  networks  was  introduced.  The  method  adds  intuitive  global  in¬ 
sight  to  the  often  fragmented  insight  of  classical  approaches  and  is  useful  for  CNN 
template  analysis. 
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Bbstract.The  following  results  have  been  obtained  on  the 
stability  analysis  of  Cellular  Neurol  Netuiorks(CHNs) . 

i) By  properly  defining  the  Lyapunov  function  for  the  CNN  we  have 
proved  that  CNNs  with  opposite-sign  template  structure  are 
completely  stable  for  certain  values  of  template  values. 

ii) By  using  the  'ToepI itz-Tridiogonal' structure  of  the  state 
matrices  of  CNNs  with  positive-cell  linking  property  we  have 
given  some  conditions  on  the  template  values  such  that  all 
trajectories  of  CNNs  converge  to  stable  equilibria. 

iii) Uhiie  investigating  the  conditions  for  the  existence  of 
equilibrium  points  of  CNNs, we  have  obtained  a  necessary  and 
sufficient  condition  for  the  existence  of  an  equilibrium  point 
in  each  saturation  region,partial  ly  saturation  region  and  linear- 
region  in  which  the  CNN  operates.Rctuai ly  this  condition  is 
voiid  not  only  for  CNNs, but  also  for  more  general  types  of 
continuous-time  neural  network  models.Final I y, stable  and 
unstable  equilibrium  points  have  been  determined. 

I. Introduction 

Rn  analog  type  neural  networks  "CNN"  with  its  local 
interconnection  properties  have  found  many  application  areas  and 
there  have  been  many  works  on  the  stability  onalysis  of  CNN's 
[1-6]  .In  [1],  Chua  and  Vang  proved  that  o  network  with 
symmetric  weight  matrix  is  completely  stable  (i.e.,  every 
trajectory  tends  to  an  equilibrium  state).  In  [2], Chua  and  Roska 
attempted  to  enlarge  the  stable  universe  of  CNN's  by  ovoiding 

"This  work  was  supported  in  part  by  the  Dept  .Electrical 
Engineer ing,iCathoi  i eke  Universiteit  Leuven, Belgium. 
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the  restriction  syinmetry  on  the  weight  matrix  such  as  opposite- 
sign  templates  with  p>l  and  conjectured  that  this  type  of  CHN's 
are  completely  stable. But  Zou  and  Nossek  ,in  [3]  proued  that  for 
some  template  values  (s>p-l)  CNN’s  with  opposite-sign  templates 
have  not  equilibrium  points  in  the  saturation  regions  which  are 
the  only  stable  regions  where  stable  equilibrium  exists.  Thus, 
they  showed  that  complete  stability  does  depend  on  the  template 
values  and  the  opposite-sign  template  structure  is  not  itself 
sufficient  for  the  complete  stabi I ity.Their  emphasize  on  the 
existence  of  equilibrium  points  in  the  stable  regions  is 
important  such  that  for  the  symmetric  weight  matrix  case  the 
existence  of  equilibrium  points  in  the  stable  regions  have  yet 
to  be  proven. 

By  defining  the  Lyapunov  function,  the  conjecture  "the 
CNN  with  opposite  sign-template  with  s<p-l  is  completely  stable" 
given  in  [3], can  be  easily  proved  since  s<p-l  already  implies 
the  existence  of  equilibrium  in  the  saturation  regions. For  the 
state  mcdel  of  the  form 

0 

X=-X  +  flf(X)  (1) 
where  fi  has  the  form  as 


after  defining  the  energy  function  as 

1 

E(t)  - - iUM)  [fl-Ilf(X) 


where  f(X)=[f(xi)  f(x2) . . . fCx^)]*  ,  X=(x, ,  X2,...,  x^)*  and 
f(xj)  is  0  piecewise  linear  function  as  defined  in  [1]  then  it 
can  easily  be  shown  that 


(p-1 ) 


E(t)  = 


E  f‘(Xi) 
1»1 
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(p-l; 

which  is  bounded  (-i.e.,  lE(t)|<  - .n  since  lf(x;)KO. 

« 

2 

The  rate  of  change  of  energy  can  be  found  as 


dE(t) 

(p-1 ) 

df(xj)  dXj 

7  ''  f  i'  ^ 

dfCxj)  n  |xil<l 

!  A  L 

dt 

2 

dxj  dt 

dXj  (o  IXjUl 

which  is  indeed  equal  to 

dE(t)  n  dxj 

-  =  -(p-1)  I  Xj  -  for  IxjUl  (3) 

dt  1=1  dt 

From  equ.(l)  Xi*(p-l)xj+sXj_| -sXj+j  and  by  subslituing  this  into 
equ. (3) 

dE  n 

-  =  -(p-1)^  I  Xj2(t)  ^  0  Vy-i 

dt  i»1 

is  obtained. 

2.S0ME  REMARKS  ON  THE  STReiLITV  RNRLVSIS  OF  POSITIUE-CELL 
LINKING  AND  OPPOSITE-SIGN  TEMPLATES 

The  renorks  given  in  this  section  is  based  on  the 
eigenvalue  foraula  of  ToepI itz-tridiogonai  lotrices  given  by 
U.F. Trench  [7].  In  [2]  the  state  eodel  (t)  sith 

P  8 

r  p  s  0 

fl  =  r*  P  ^  ,  r,s  >  0  (1) 

6  ‘  s  p  >  1 

r  p_ 

Is  considered  and  by  nenark  3  it  eas  proved  that  the  solution 
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of  3uch  a  dynamical  system  ,  for  almost  ali  initial  conditions 
conuenges  to  stable  equilibria  which  are  in  the  saturation 
regions.  This  result  is  the  direct  consequence  of  the  theorem 
(•f.l)  giuen  by  Hirsch  in  [8]  . Actual  ly  the  system  (n)  with 
r.s>0  defines  a  completely  stable  system  according  to  the 
theorem  giuen  by  J.  Smillie  [9]  . 

In  the  linear  region, aboue  system  corresponds  to  the 


0 

X 


P-1 


r 

9 


s 


9 

‘  c  s 
& 

r  p-1 , 

nxn 


and  its  ei genua  lues  are 

qir 

»(p-1)+2  4  r.s  cos( - ),  Kqin  (5) 

n+1 

i)  p>1  ensures  that  there  exists  a  posit iue  eigenualue  and  hence 
almost  all  solutions  tend  to  leaue  the  linear  and  partial 
saturation  regions  towards  to  the  saturation  regions. 

ii)  If  we  consider  case  I  "i.e.,  (p-1  )>2*rrTs,  r.8>0  then  all  the 
eigenuaiues  are  posit  iue  and  all  the  solutions  of  system  (*1) 
tends  to  complete  saturation  regions. 

iii)  Considering  case  II  "i.e.,  (p-1)<24Frs  ,  r.8>0,  p>i  and 

w  (p-1) 

Co8( - )  <  -  ■  then  all  the  eigenuaiues  are 

n*1  2r8 

positive  and  all  the  solutions  converge  to  the  stable 
equilibria  in  the  saturation  regions  if  they  exist, 
iu)  Case  III  implies  that  almost  all  solutions  will  leave  the 
linear  and  partial  saturation  regions  even  if  d<1  since 
there  exists  a  positive  eigenvalue. 

The  above  argument  shows  that  by  arranging  the  template 
values  we  can  obtain  a  completely  stable  system  such  that  all 
the  solutions  of  (4)  converge  to  the  equilibrium  points  in  the 
saturation  regions  and  case  (iv)  also  shows  that  almost  all 
solutions  converge  to  the  stable  equilibrium  points  in  the 
complete  saturation  regions  even  if  d<1  as  we  already  know  that 
p>1  is  not  a  necessary  condition  for  the  stable  equilibrium 
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points  being  in  the  saturation  regions  but  only  a  sufficient 
condit ion. 

The  foraula  (5)  aiso  confirns  the  results  obtained  by 
Roska  that  the  real  part  of  all  the  eigenvalues  of  system  (1) 
mith  r=-s“  i.e.,  opposite-sign  template  system"  in  the  linear 
and  partial  linear  regions  are  strictly  positive  since 

qi 

?wq*(p-1)+jscos( - ) 

n*l 

and  p>l . 

S.Conditions  For  EHistence  of  Equilibrium  Points 

Let's  consider  the  dynamic  equation  (1).  The  external 
inputs  are  not  considered  because  the  system  vith  constant  input 
can  alotays  be  transformed  into  this  form  by  the  method  given  in 
[10]. 

Theorem! ;  There  exists  an  equilibrium  point  in  each  of  3^ 
regions  (including  I inear,part iai  soturation  regions  and 
complete  saturation  regions)  if  and  oniu  if  the  matrix  fl-I  is 
diagonal ly  dominant . 

Coro  I  lory  :  If  there  exists  a  rom  (Index  L)  such  that 
n 

a^-l<-  2  lai.jl  (mhlch  indeed  implies  (6) 

then  there  doesn't  exist  any  equilibrium  point  in  ony  saturation 
region. 

note  1;  In  [11  ]j  it  has  been  proved  that  if  inequality  is 
satisfied  for  every  l■1,2...n  (i.e.,  if  "I-fl"  is  strictly 

diagonal  dominant  eith  0|;<1)  then  there  exists  unique  stable 
equi librium  point , 

Conjecture  ;  If  fl-I  is  strictly  dominont  then  a  trajectory 
corresponding  to  the  system  (1)  converges  to  an  equilibrium  In 
one  of  the  soturation  regions  (i.e.,  the  systems  (1)  is 
completely  stable). 
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Abstract.  The  problem  of  using  dynamical  equilibria  of  a  Cellular  Neural 
Network  (CNN)  of  relaxation  oscillators  is  faced.  The  behavior  of  a  planar 
matrix  and  of  a  two-torus  of  connected  oscillators  is  investigated  by 
computer  simulation.  Some  theorems  are  stated  in  the  case  of:  i)  two 
oscillators,  ii)  a  tree  of  oscillators,  iii)  a  loop  of  oscillators. 

Introduction.  It  is  well  known  that  among  neural  networks,  CNN's  [1] 
ejdiibit  the  advantage  of  possessing  only  local  connections.  This  property 
is  important  with  respect  to  their  implementation  by  means  of  dedicated 
integrated  circuits. 

On  the  other  hand,  scane  recent  studies  put  into  evidence  that  many 
biological  systems  behave  as  distributed  oscillatory  systems  i.e.  large 
arrays  (generally  planar)  of  coupled  oscillators.  A  good  conjecture,  based 
on  results  of  experiments  perfonned  on  the  visual  cortex  of  cats  and  the 
olfactory  cortex  of  rabbits  (2,3J ,  seems  to  be  that  synchronization  on 
different  attractors  emerging  from  a  chaotic  substratum  could  be  an 
efficient  way  of  storing  and  processing  information. 

These  two  facts  induced  us  to  start  studying  a  new  architecture 
possessing  these  two  distinctive  features: 

1)  Its  topology  is  similar  to  that  of  a  planar  Cellular  Neural  Network 
with  connectivity  1.  but  with  links  only  along  rows  and  columns  of  a  matrix 
(no  diagonal  connections) ; 

2)  Neurons  are  oscillators  instead  of  nonlinear  controlled  sources.  We 
chose  to  use  relaxation  oscillators  [4J  for  two  reasons:  i)  Coi^xiter 
simulations  can  result  more  exact  as  in  this  case  integration  is 
straightforward;  ii)  Hardware  implementation  can  take  advantage  of  using 
only  digital  circuits. 

Therefore  we  study  a  matrix  of  connected  oscillators,  the  behavior  of 
vrfiich  is  of  course  dependent  on  many  parameters:  the  natural  frequency  of 
each  oscillator,  the  strength  of  each  coupling,  the  initial  state  of  the 
system.  As  a  natural  extension,  we  will  consider  also  the  same  matrix  wound 
on  a  two-tonis,  even  if  the  analogy  with  biological  structures  becomes  less 
evident  in  this  case. 


0-7803-0875-1/92  3$.00  01992IEEE 
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At  our  best  knowledge  it  is  possible  to  find  in  literature  only 
studies  on  the  existence  of  periodic  attractors  for  systems  of  sinusoidal 
oscillators  with  random  natural  frequencies  and  equal  coupling  coefficients 
[5,6]. 

In  the  following,  we  shall  denote  as  the  strength  of  the  coupling 
between  oscillator  i  and  j,  i.e..  the  amplitude  of  the  signal  arriving  to 
oscillator  i  fran  oscillator  j.  determines  the  natural  frequency  of 
oscillator  i. 

Experimental  results.  IXie  to  the  amount  of  parameters  we  have  to  deal 
with,  that  is  very  large  even  in  the  case  of  a  small  (4  by  4)  matrix,  as  a 
first  step  we  started  using  the  so  called  "experimental  mathematics". 
Therefore  we  performed  the  following  experiments: 

A)  We  considered  a  planar  matrix  of  16  identical  oscillators  (i.e. 
having  the  same  natural  frequency)  and  found  the  following  types  of 
behavior : 

1)  In  the  case  when  the  coupling  coefficients  are  all  equal  to  each  other, 
the  behavior  of  the  system  is  chaotic  for  every  initial  state.  The  Lyapunov 
exponent,  computed  by  following  trajectories  in  state  space  and  applying 
Wolf's  algorithm  [7].  seems  to  be  everyvrfiere  positive,  except  when  starting 
from  the  origin  of  state  space,  where  Lyapunov  exponent  is  zero. 

2)  In  the  case  vhen  the  coupling  coefficients  satisfy  condition 

sign(k.  .)-*-3ign(k.. ) 

(opposite-sign  condition),  the  system  always  reaches  a  stable  oscillatory 
state  at  the  common  natural  frequency  of  the  oscillators,  with  every 
oscillator  differing  in  phase  for  1/4  period  from  its  neighbors  (obvious 
consistency  conditions  apply) .  Magnitude  of  the  coefficients  is  not 
important . 

The  stabilizing  effect  of  the  oppos:  sign  structure  is  analogous  to 
the  case  of  ordinary  Cellular  Neural  Networks  [8]. 

3)  With  random  coupling  coefficients,  different  stable  limit  cycles  are 
observed,  together  with  non-periodic  evolutions:  quasi-periodic  or  chaotic. 

In  order  to  inquire  into  the  structure  of  the  attraction  basins  of  the 
said  limit  cycles,  we  performed  simulations  taking  initial  conditions  along 
segments  in  the  16-dimensional  state  space.  Structures  of  these  sets  look 
not  at  all  trivial,  and  transitions  between  neighboring  basins  often  seem 
to  be  fractal.  However,  the  presence  of  a  set  of  clearly  distinct  stable 
limit  cycles,  with  attraction  basins  possessing  at  least  a  closed  inside, 
makes  us  believe  that  they  might  lend  themselves  to  be  used  as  "codes"  or 
"memory  states"  for  a  content-addressable  memory  (CAM) . 


247 


The  overall  structure  of  attractors  is  more  dependent  on  the  mutual 
relations  t»etween  coupling  coefficients  (e.g.  sign,  topology...)  than  on 
their  magnitude.  In  next  section  this  is  proved  on  simpler  structures. 

Another  set  of  experiments  was  performed  in  order  to  study  the 
transition  between  a  situation  when  a  stable  limit  cycle  is  observed  (case 
3)  and  that  described  under  case  1  (transition  to  chaos) .  For  this  purpose, 
the  evolution  of  the  network  was  simulated  for  different  values  of  its 
parameters  obtained  by  moving  along  a  segment  in  the  48-dimensional  space 
of  free  coefficients.  The  results  of  one  of  these  simulations  are  shown  in 
figure  1,  where  the  ordinate  is  the  rvDnDalized  frequency  attained  at  the 
end  of  the  transient.  Zero  ordinate  means  non-periodic  behavior. 

B)  We  cilso  cor.sidered  matrices  on  a  two-torus,  which  also  showed 
trivial  stable  behavior  under  the  opposite-sign  condition. 

In  the  case  of  identical  coefficients,  toroidal  matrices  show  a  much 
larger  number  of  periodic  states,  with  smaller,  most  probably  zero-measure, 
attraction  basins.  Two  kinds  of  behavior  were  observed,  the  prototypes  of 
which  ai'e  shown  in  figures  2  and  3  Where  the  abscissa  is  taken  over  a 
segment,  in  the  16-dimensional  space  of  initial  states,  joining  two  proper 
sets  of  them.  Figure  2  shows  continuous  change  among  indifferent 
equilibrium  limit  cycles,  while  figure  3  shows  complex  kneading  of  periodic 
and  non-periodic  evolution.  As  an  example  of  instability,  the  leftmost 
state  of  figures  2  and  3  (the  same  in  the  two  cases)  is  periodic,  but 
Lyapunov  exponent  estimation  gives  a  value  of  about  1.2. 

In  the  case  of  coefficients  all  different  from  each  other,  randomly 
chosen,  no  periodic  evolution  has  ever  been  observed. 

Theoretical  results.  We  shall  consider  in  tlnis  section  simpler  structures 
than  matrices,  in  increasing  order  of  complexity.  Most  of  thie  results 
described  can  be  straightforwardly  extended  to  sinusoidal  oscillator 
systems . 

1)  Thie  simplest  system  consists  of  two  mutually  coupled  oscillators. 
Such  a  system  shows  a  wide  spectrum  of  behavior's:  (i)  single  stable  limit 
cycle,  with  oscillators  operating  all  at  the  same  or  (ii)  at  different 
freqijencies;  (iii)  periodic  evolution  at  a  frequency  dependent  on  initial 
conditions:  (iv)  quasi-periodic  or  non-periodic  motion. 
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The  following  results  are  proved  in  [9]: 

Theorem  1:  ftecessary  and  sufficient  condition  for  the  two  oscillators 
to  evolve  at  the  same  frequency  is  that 

Ik  -  k  I  >  |k  -  k  I 
12  21  '  .11  22 

k  k  ^  k  k 
22  12  11  21 


Duty  cycle  is  50%. 

Theorem  3:  If  k  A  “  k  =•  r.  then  the  evolution  of  the  system 
is  periodic  if  r  is  rational,  else  quasi-periodic,  and  the  rate  of  the 
frequencies  of  the  two  oscillators  is  r. 


Theorem  3:  If  k 


k  and  k 
22  12 


(bifurcation  condition)  the 


systems  evolves  periodically  with  no  transitory,  because  there  is  an 
infinity  of  periodic  indifferent  equilibrium  states. 

2)  A  more  complex  system  is  a  tree  of  oscillators.  For  such  a 
structure  it  is  possible  to  canpute  easily  a  linear  relation  between  the 
frequencies  of  the  oscillators.  The  algcarithm  is  described  in  [9].  It  can 
be  extended  to  a  general  structure,  where  also  loops  are  present,  but  the 
relation  becomes  nonlinear,  and  conpatational  difficulty  increases 
rapidly.  The  following  results  are  proved  in  [9]: 


Theorem  4:  Let 


1  -  E  R  ■D’^ 

j€<t> 


-  S ,  vi 

k€<j> 


R  k. .  X  k.. 
'•a  '■j  ji 


If  all  oscillators  evolve  at  the  same  requency  f.  and  O.then  f  can  be 
ccmputed  through  the  following  irecurrent  expression,  starting  from  any 
oscillator  i  of  the  tree; 


f  -  0.5  N®  /  D° 


vrtnere 


N°  -  k. .  -  E  R  n" 

»•  .  .  '■J  J 

j€<v> 

-  k..  -  E  Ru  N;* 

J  JJ  .  .  Jie  k 

ke<j> 

k»«i 

and  <i>  is  the  set  of  indices  of  all  oscillators  connected  to  oscillator  i. 
If  all  oscillators  have  the  same  natural  frequency  and  D^-0 
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(bifurcation  condition)  then  (1)  takes  the  form  0/0,  and  so  loses 
significance;  multiple  attractors  are  present  (as  in  the  case  of  two 
oscillators),  depending  on  initial  conditions. 

When  a  loop  is  considered,  the  bifurcation  condition  may  be  stated  as 
the  bifurcation  condition  on  two  open  chains  obtained  from  it. 

It  is  reasonable  to  conjecture  that  the  bifurcation  condition  for 
matrices  corresponds  to  the  fact  that  any  tree  that  can  be  extracted  from 
the  matrix  is  in  the  bifurcation  condition. 

Conclusion.  A  planar  array  of  relaxation  oscillators  has  been  considered  in 
the  view  of  using  it  as  a  new  basic  structure  in  the  field  of  Neural 
Network  architectures.  Although  this  goal  has  not  yet  been  attained,  such  a 
structure,  never  studied  before,  exhibits  a  rich  and  interesting  behavior. 
A  systematic  research  about  it  was  started  and  some  basic  results  were 
obtained. 
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Fig.  1  -  Normalized  period  vs.  normalized  abscissa  on  a  segment 

(48-dimensional  space)  joining  two  different  sets  of  coefficients,  one 
corresponding  to  periodic  behavior,  the  other  to  chaos  (bottom  ordinate) 


Fig.  2  T  Normalized  period  vs.  normalized  abscissa  on  a  segment  joining  two 
different  initial  conditions  in  state  space.  Infinite  periodic  attractors 
of  zero  measure  are  shown. 


Fig.  3  -  Normalized  period  vs.  normalized  abscissa  on  a  segment  joining  two 
different  initial  conditions  in  state  space.  Mixed  behavior  of  chaos 
(bottom  ordinate)  and  periodic  oscillations  is  shown. 
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abstract 

In  this  paper,  Discrete-time  Cellular  Neural  Networks  using  digital 
neuron  model  with  DPLL  (digital  phase-locked  loop)  is  proposed.  Since 
consists  of  all  digital  elements,  it  is  easy  to  realize  the  neuron  model  by  VLSI. 
We  show  theoretical  analysis  and  apply  it  to  Shadow  Detector  in  [3]  in  order 
to  confirm  the  action  of  the  our  neuron  model  by  the  computer  simulation. 

Introduction 

Cellular  Neural  Networks  [1][2]  represent  a  new  paradigm  for 
nonlinear  analog  signal  processing.  In  particular.  Discrete-time  cellular 
neural  networks  (DTCNN)  have  been  introduced  in  [3J,  which  represent  an 
efficient  architecture  for  image  processing  and  pattern  recognition.  The 
system  can  solve  global  task,  although  the  cell  are  only  locally  connected. 
The  simple  structure  of  single  cell  and  the  local  connectivity  make  them  well 
suited  for  VLSI  implementation.  The  realization  of  DTCNN  has  been  reported 
in  [4]  .  This  has  analog  circuit  structure  designed  for  a  CMOS  process. 
However,  analog  circuits  have  the  following  weak  points.  It  is  difficult  to 
connect  neuro-chips  and  realize  modifiable  analog  synaptic  weight.  And 
the  error  performance  of  analog  circuits  are  poorer  than  that  of  digital.  It  is 
widely  expected  to  realize  the  DTCNN  by  all  digital  circuits. 

The  neuron  model  by  using  multi-input  multilevel-quantized  digital 
phase-locked  loop  (MM-DPLL)  has  been  proposed  in  [5].  This  model  has  the 
following  advantage.  Since  it  consists  of  all  digital  elements,  it  is  easy  to 
realize  the  neuron  model  by  VLSI.  This  is  the  digital  neuron  model  utilizing 
the  phase  information.  By  using  the  phase,  the  input  signal  is  modulated  by 
using  DPLL,  so  the  model  can  be  applied  to  the  various  problems.  In  [5],  it 
was  applied  to  the  Hopfield  type  associative  memory  and  the  pattern 
recognition.  This  neuron  model  satisfies  the  requirements  of  the  neuron's 
characteristics  such  as,  spatial  summation,  temporal  summation, 
thresholding  and  learning  by  variable  weight.  Since  the  phase  modulated 
signal  is  widely  used  in  field  of  the  modern  digital  communications,  the 
phase  information  type  of  the  digital  neuron  model  will  be  required  in  near 
future.  In  this  study,  we  propose  Discrete-time  Cellular  Neural  Networks 
using  digital  neuron  model  with  DPLL  and  show  theoretical  analysis.  We 
apply  it  to  Shadow  Detector  in  (3j  and  confinn  the  action  of  the  our  neuron 
model  by  the  computer  simulation. 

Discrete-time  Cellular  Neural  Networks 

DTCNN  are  defined  by  the  algorithm: 

x'^(0=  I  ^  bdu‘^+ 

deNr(c)  deNr(c)  /i\ 
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>‘'(/)=/(x‘'(/-l))=l  forx^>0 

-1  forx‘^<(i  (2) 

where  the  output  state  y^(t)  of  a  cell  c  is  binary  and  is  determined  by  the 
sign  of  x^(t-l).  The  value  of  is  controlled  by  the  inputs  and  outputs  of 
adjacent  cells  d  within  an  r-neighborhood  Nr(c).  This  is  defined  as  the  set  of 
adjacent  cells  within  a  distance  r  including  cell  c.  The  template  coefficients 
are  translational  invariant.  The  value  of  e  Hi  is  constant  and  used  for 
threshold. 

Digital  Neuron  Model  with  DPLL 

Before  fixing  the  circuit  structure,  the  grid  topology  and  the 
neighborhood  size  have  to  be  determined.  We  have  chosen  a  1- neighborhood 
on  the  rectangular  grid  as  shown  Fig.l,  where  each  cell  is  connected  only  to 
its  nine  nearest  neighbors.  In  this  case,  the  proposed  model  as  shown  in 
Fig. 2,  consists  of  eighteen  multilevel-quantized  phase  comparators(P.C), 
eighteen  random  walk  filters  (RWFi  (/  =  1~18)),  a  random  walk  filter  (RWFc), 
a  digital  voltage  controlled  oscillator  (Digital  V.C.O.).  a  binary-quantized 
phase  detector  (P.D.),  and  a  phase  sifter.  These  are  all  digital  elements.  We 
treat  the  phase  modulated  signal  as  input.  The  phase  of  input  signal  leads  (or 
lags)  the  reference  phase  by  phase  difference  6y^  and  0^^  [rad].  So  the  input 
of  this  model  can  be  expressed  as  analog  value. 

Each  multilevcl-quantized  phase  comparator  (P.C.)  compares  the 
corresponding  input  0yd  and  Syd  with  the  output  phase  of  digital  V.C.O., 
where  0(t)  is  initially  set  to  be  equal  to  the  reference  phase.  As  shown  in 
Fig. 3,  the  phase  difference  is  quantized  by  using  the  internal  high-speed 
clock  operating  at  R-times  the  loop  natural  frequency  /q. 

A  sequence  of  pulses  produced  as  lead  (or  lag)  signals  is  fed  to  up  (or 
down)  inputs  of  each  random  walk  filter  (RWFt)  in  Fig.4,  which  is  an  up- 
down  counter  of  set  length  Ni.  The  content  of  this  counter  is  increased  by  up 
inputs  and  decreased  by  down  inputs.  When  the  content  reaches  2Ni  or  0,  an 

erase(-l)  or  add  (+1)  signal  is  obtained  as  the  output  respectively,  and  the 
counter  is  simultaneously  reset  to  Ni.  In  short,  RWFt  divides  the  sequence  of 

pulses  by  Ni  and  acts  as  the  constant  multiplier  of  weight  and  =  1  /Ni. 

The  sequence  of  pulses  weighted  by  and  from  RWFi  is  summed 

spatially  as  follows, 

deNr(c)  deNrf^c)  (3) 

and  the  spatial  summation  is  sent  to  the  second  RWFc  (set  length  Nc)  which 
keeps  the  output  of  digital  V.C.O.  stable  and  provide  erase  (-1)  or  add  (+1) 
signals.  The  second  RWFc  is  inserted  because  a  large  number  of  pulses 
control  the  output  phase  0(t)  excessively. 

Digital  V.C.O.  consists  of  three  parts  :  fixed  frequency  oscillator.  Phase 
controller  and  divider.  As  shown  In  Fig. 5,  the  phase  controller  eliminates 

one  pulse  for  an  erase  signal  or  adds  one  pulse  for  an  add  signal  with  the 

fixed  frequency  oscillator.  The  output  of  phase  controller  is  divided  by  R  to 
provide  the  output  signal  synchronized  to  the  sum  of  the  weighted  input 
phase  signals. 

Binary-quantized  phase  detector  (P.D.)  compares  0(t)  with  threshold  h 
of  the  phase  decided  by  phase  sifter.  When  0(t)  exceeds  h,  the  output  of  our 
neuron  model  is  y  =  1,  otherwi.se  it  is  -1, 
y  =  SGN  {0  -  h)=  ]  0>  h 

-\  0<h  (4) 

Our  model  can  be  analyzed  by  linear  model  shown  in  Fig. 6  because  the 
phase  comparison  characteristic  is  linearly  approximated  at  large  R  ( 
dividing  ratio)  ,  where 
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RI2n  :  phase  comparison  coefficicnl 

~  ■  cocITicicnl  of  RWFi 

UNc  :  coefficient  of  RWFc 

InfQlR  :  coefficient  of  digital  V.C.O. 

/q  :  the  input  frequency 

The  linear  differential  equation  of  this  model  is  expressed  as 

I  1  ^  I 

f(f),  _ deNrje) 

I  .  I  bl 
1  deNr(c)  ^^Rr(c) 


7-5-1  i  5^  K 


‘''c\deNr(c)  deNr(c)  I 

(t>(t)  is  obtained  as 

j  ^  a§  y‘‘  *  I 

'  ^  1  aS  ^  1  b^ 

deNr(c)  deNr(c)  I 

At  the  stationary  state  of  the  step  response,  Eq.(7)  becomes 

/  T  «  C  ..W  .  V 


0(0  = 


In  Eq.(6),  t  is  decided  by  the  input  frequency  /q  ,  set  length  Nc  of  second 


I  a§  yd 

+  I 

Nr(c) 

de  Nr{c) 

1  ai 

+  1  b 

deNr(c) 

deNr(c) 

RWFc  and  the  sum  of  weight.  By  solving  t  ,  we  can  obtain  the  time  that  a 
neuron  reaches  the  stationary  state  and  set  up  a  discrete  time  step. 


Simulation 

In  the  simulation  to  confirm  the  action  of  the  our  neuron  model,  we 
use  Shadow  Detcctor[4].  The  two-dimensional  output  pattern  should  consist  of 
the  shadow  created  by  an  illuminated  input  pattern.  Accepting  the 
restriction  of  a  light  source  only  at  the  right  boundary  of  the  grid,  the  task 
can  be  solved  by  a  wave  propagation  that  starts  there.  The  initial  state  is 
black  and  the  pattern  is  used  as  input  for  the  cells.  The  wave  propagation  in 
the  form  of  transitions  from  black  to  white  starts  from  the  right  boundary 
and  stops,  if  a  black  pixel  in  the  input  pattern  is  reached.  The  remaining 
black  pixels  correspond  to  the  shadow.  Set  length  of  RWFt  and  weight  set  in 
all  neuron  are  as  follows: 


Ni  N2  N3  00  00  00 

N4  N5  =  00  32  16 

N7  Ns  Nc;  00  00  00 


Nj(jN]i  Nj7  00  00  00 

N]3  N)4  Ni5  =  00  16 

N]6  N)7  N]s  00  00  00 


0  0  0  0  0  0 
a-  0  1/32  1/16  b=  0  1/16  0 

0  0  0  0  0  0  (9) 

The  input  signal  is  modulated  by  0^^  and  0^}  =  +7t/2  [rad]  at  the  black  region  ( 

y^,  =  1),  and  the  signal  modulated  by  0y^  and  6^^  =  -nil  [rad]  at  the  white 

region  (  y^,  =  -1).  When  the  input  frequency  (/q)  is  lOOkHz  and  Nc  =  32, 

we  considered  one  cycle  to  0.01  [msj.  Fig. 7  shows  an  input  pattern  and  the 
output  states  in  0,2,4,6,8,10  cycle.  From  this,  we  can  find  that  our  digital 
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neuron  model  has  the  correct  action.  The  result  shows  that  MM-DPLL  can  be 
applied  to  the  DTCNN. 

Conclusions 

Our  neuron  model  consi.sts  of  all  digital  elements.  DTCNN  has  the  simple 
structure  and  local  connectivity.  Discrete-time  Cellular  Neural  Networks 
using  Digital  Neuron  Model  are  suitable  for  VLSI  implementation.  It  is 
possible  to  realize  the  architecture  of  Discrete-lime  cellular  Neural  Networks, 
in  which  the  internal  information  is  the  phase. 
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Fig.l:  Neighbor  cells  of  cell  c  for  r  =  1. 


Fig.2:  Block  diagram  of  the  MM-DPLL 
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Fig.3:  Timing  chart  of  mulilevel-quantized  phase  comparator 
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Fig.5:  Timing  chart  of  digital  V.  C.  O. 
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Fig.6:  Linear  loop  model  of  the  MM-DPLL 


Fig.7:  Shadow  Detection 
Inpul  pallem  and  oulpul  slates  in  our  simulation 
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Abstract 

Symmetry  properties  of  cellular  neural  networks  on  square  and  hexagonal  grids  are  investi¬ 
gated:  While  the  symmetry  inherent  in  a  given  problem  should  be  reflected  in  the  structure  of  a 
CNN  designed  to  solve  it,  the  degree  of  symmetry  possible  in  the  network  is  limited  to  the 
symmetry  group  of  the  grid  it  is  defined  on.  The  exact  numbers  of  independently  choosable 
entries  in  both  square  and  hexagonal  CNN-templates  with  different  important  kinds  of  symme¬ 
try  are  compared.  As  expected,  the  higher  symmetry  of  the  hexagonal  grid  leads  to  a  significant 
reduction  in  the  complexity  of  the  templates. 


. )  Introduction 

in  technology,  sampling  of  spatial  information  is  done  on  square  grids.  Nature,  on  the  other 
hand,  organizes  its  information  processing  elements  in  hexagonal  arrays.  Although  the  evolu- 
r'onary  success  of  this  arrangement  is  probably  due  to  the  fact  that  it  corresponds  to  the  densest 
packing  of  circles  in  the  plane,  which  arises  naturally  during  cell  growth  on  surfaces,  it  may 
also  simply  be  better  suited  for  many  problems  because  of  its  higher  degree  of  symmetry. 

in  Cellular  neural  networks  or  CNNs  as  introduced  by  Chua  and  Yang  [1],  processing  ele¬ 
ments  are  also  often  arranged  in  a  regular  grid,  and  programming  for  a  particular  task  is  done  by 
.  boosing  a  set  of  parameters  which  are  arranged  as  a  regular  template.  As  will  be  shown, 
^symmetrical  tasks  pose  mote  stringent  constraints  on  hexagonal  templates  than  on  square  ones, 
in  practice,  this  means  that  hexagonal  systems  may  be  easier  to  build.  Symmetry  may  also  be 
Introduced  as  a  design  constraint  in  a  more  direct  design  method  such  as  the  one  proposed  in 

[ll 

2)  Coordinate  Systems  and  Neighbourhoods 

On  both  grids,  coordinate  systems  must  be  introduced.  To  eliminate  boundary  problems,  the 
grids  are  supposed  to  be  infinite  and  plane-tilling. 

Let  5^  be  the  square  grid  with  the  cartesian  coordinate  system,  as  shown  in  Fig.  1.  Individual 
cells  are  denoted  by  C(i,  j),  where  i  is  the  row  index  and  j  is  the  column  index. 

The  r-neighbourhoodNI(i,  j)  of  a  cell  C(i,  j)  in  5^,  r  e  N  being  a  positive  integer  and  the 
superscript  s  denoting  the  square  grid  is  defined  as  in  [1]  as  the  following  set  of  cells: 

N?(i,  j)  ;=  {C(k,  1)  I  max  (Ik  -  i|,  |1  -  jl}  ^  rj.  (1) 

In  order  to  simplify  notation,  we  will  also  allow  neighbourhoods  of  coordinate  pairs. 
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Fig.  1 :  The  square  grid  (left)  arid  hexagonal  grid  (right). 


Let  be  the  hexagonal  grid,  with  the  hexagonal  coordinate  system  as  shown  in  Fig.  1. 
Cells  are  again  denoted  by  C(i,  j),  where  i  is  the  row  index  and  j  is  the  "slanted"  column  index. 

The  (hexagonal)  r-neighbourhoodNi(i,  j)  of  C(i,  j),  where  the  superscript  h  indicates  the 
hexagonal  grid  is  the  set  of  (coordinates  of)  all  cells  whose  centers  may  be  connected  to 
the  center  of  C(i,  j)  by  a  curve  that  intersects  at  most  r  hexagon  boundaries. 

As  soil  has  the  structure  of  a  lattice,  it  can  be  slanted  such  that  the  centers  of  its  cells  fall  on 
a  square  grid.  The  image  of  Nf(i,  j)  under  this  transformation  is  the  subset  of  N?(i,  j)  that  ex¬ 
cludes  all  cells  in  the  top  left  and  bottom  right  comas  whose  manhattan  distance  to  the  center  is 
larger  than  r.  Therefore  Nj*(i,  j)  is  defined  analytically  as: 

N?(i,  j)  :=  {C(k,  1)  I  max  {|k  -  i|,  |1  -  j|,  |(k  -  i)  +  0  -  j)|}  ^  r).  (2) 


3)  Symmetrical  Problems 

Symmetry  is  the  invariance  of  an  object  under  a  group  of  actions.  The  symmetry  groups  of 
most  image  processing  problems  are  continuous  Lie-groups.  In  electrical  engineering  systems, 
however,  image  data  is  sampled  before  being  processed  on  a  discrete  grid,  which  leads  at  best 
to  a  discrete  subgroup  of  the  original  Lie-group.  Still,  the  symmetry  inherent  in  an  image 
processing  problem  should  be  reflected  as  closely  as  possible  by  structural  symmetry  of  the 
processing  array. 


3.1)  Space  Invariance 

Let  M  be  some  space  and  T(M)  the  largest  group  of  translations  that  leaves  the  structure  of  M 
unchanged.  Then  any  image  processing  problem  posed  in  M  that  is  invariant  under  T(M)  is 
space-invariant. 

The  translation  group  T(R^)  of  the  real  plane  is  simply  given  by 

T(R^)  =  (tr:  R^->  R^|  afxi,  X2)  =  (xi  +  Axi,  X2  +  AX2)  a  Axi,  Ax2  6  r).  (3) 
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The  translation  groups  T(5^  and  are  isomorphic  to  T(Z^),  the  restriction  of  T(R^)  to 

the  integer  lattice  Z^,  which  performs  only  integer  offsets: 

T(Z^)  =  (tr:  Z^  -4  Z^  I  tr(i,  j)  =  (i  +  Ai,  j  +  Aj)  a  Ai,  Aj  6  z).  (4) 

Space  invariance  of  the  connection  coefficients  a*^’  *  j  implies: 

=  ViJeZ  A  treT(Z\ 

=»  a'^  'i,j  =  a"""’ V  Ai,  Aj  €  Z.  (5) 

The  connection  coefficients  in  space  invariant  CNNs  therefore  depend  only  on  the  cells’  relative 
positions.  This  allows  a  description  in  terms  of  relative  positions,  the  templates  defined  in  [1]: 

a(Ai,Aj):=a‘’^j^^j^.^^.  V  i,  j,  Ai,  Aj  €  Z.  (6) 

If  interactions  are  local,  a  unique  minimal  Neighbourhood  Nr^(0,  0)  of  the  template’s  origin  (0, 
0)  with  minimal  finite  rmin  exists,  which  contains  all  non-zero  coupling  coefficients: 

rmin  =  min  (r  I  {a(i,  j)  s  0  V  (i,  j)  €  Nr(0, 0))).  (7) 

A  template  a(i,  j)  whose  elements  are  only  specified  for  (i,  j)  €  Nr(0,  0)  (and  which  is  assumed 
to  be  zero  elsewhere)  is  called  an  r-template. 

The  numbers  ns(r)  and  n**(r)  of  entries  in  square  and  hexagonal  r-templates  are  simply 
n*(r)  =  lNf(0,  0)1  =  (2r  +  1)2  =  4r2  +  4r  +  1.  (8) 

nh(r)=|NK0,0)|=l+6X  i=  1 +3r(r+ l)  =  3r2  +  3r-H.  (9) 

i=l 

For  large  r,  a  hexagonal  r-template  contains  25%  less  elements  than  a  square  one  for  the  same  r. 

3.2)  Isotropy 

Let  M  be  some  space,  and  R(M)  the  largest  rotational  symmetry  group  of  the  neighbourhoods 
defined  in  M.  Then  any  space  invariant  problem  defined  on  M  that  is  also  invariant  under  R(M) 
is  isotropical. 

The  n-element  rotation  group  Cn  consists  of  all  n  rotations  about  integer  multiples  of  ^  360°, 

c„  =  (e,c<'>,c“>.....c<”-'>1,  (10) 

where  E  is  the  unit  element  and  the  smallest  clockwise  rotation  generates  the  group: 

COH-i).=  Ci,i)oC<j),ie  {l,2,....n-l}.  (11) 

=  (12) 

Group  theory  proves,  that  the  invariance  of  any  object  under  the  generators  of  a  group  implies 
its  invariance  under  the  whole  group.  The  invariance  of  a  square  template  under  its  rotational 

symmetry  group  R(N?(0,  0))  =  C4  is  consequently  implied  by  its  invariance  under  C4  \  whose 
action  is  defined  by 

C^4^a(i,j)-^a(j,-i).  (13) 
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(14) 


Thus,  a  necessary  and  sufficient  condition  for  isotropy  of  square  r-templates  is 

a(i,  j)  =  a(j,  -i)  V  (i,  j)  6  Nf(0,  0). 

As  (0,  0)  is  the  only  fixed  point  of  d'^\  and  ali  other  template  entries  are  mapped  on  and  are 
therefore  identical  to  exactly  three  others,  the  number  nf  (r)  of  independently  choosable  elements 
on  isotropical  square  r-templates  is  given  by 

nf(r)=l+^^^^^^^  =  r2  +  r+l.  (15) 

Similarly,  a  hexagonal  template  is  invariant  under  its  rotational  symmetry  group  R(Nj*(0,  0))  = 
Ce,  if  and  only  if  it  is  invariant  under  whose  action  in  is  defined  by 

Cg  a(i,  j)  a(i  +  j,  -i).  (16) 

A  necessary  and  sufficient  condition  for  isotropy  of  hexagonal  r-templates  is  therefore 

a(i,  j)  =  a(i  +  j,  -i)  V  (i,  j)  e  N}?(0,  0).  (17) 

As  (0, 0)  is  the  only  fixed  point  of  and  all  other  template  entries  correspond  to  five  others, 
the  number  n^(r)  of  independently  choosable  entries  in  isotropical  hexagonal  r-templates  is 

„|.(r)  =  l+!M:LL=i+2r?^  =  £^+l.  (18) 

6  o  2 

For  large  r,  isotropical  hexagonal  r-templates  contain  only  about  half  as  many  independent 
entries  as  squai'e  ones. 


3.3)  Complete  Symmetry 

Let  M  be  some  space,  and  S(M)  the  (maximal)  symmetry  group  of  the  neighbourhoods  defined 
in  M.  Then  a  space  invariant  problem  on  M  that  is  also  invariant  under  S(M)  is  completely 
symmetrical. 


The  symmetry  group  Cnv  of  a  regular  n-sided  polygon  consists  of  Cn  and  n  reflections: 


C„v  =  Cn 


(19) 


If  n  is  even,  the  reflections  are  defined  by  the  orientation  of  their  axes  as  follows: 


•  All  axes  contain  (0, 0), 


•  is  horizontal, 

•  o^'  is  obtained  from  by  tilting  it  ^  180®  clockwise  around  (0, 0). 
By  construction,  Cnv  is  shown  to  be  generated  by  and  First, 


(20) 


which  generates  the  subgroup  Cn  of  Cnv  as  shown  in  (10)  to  (12).  The  other  reflections  arc  then 
obtained  as  follows: 


(21) 
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<42i^i):=c4‘)oo<„^)o(4"-‘>.  (22) 

Hence,  a  template  is  completely  symmetrical  iff  it  is  invariant  under  (4*^^  and  a|,^^  on  its  grid. 

The  symmetry  axes  of  square  templates  are  shown  in  Fig.  2.  The  actions  of  and  on 
square  templates  are  defined  by: 

af:a(i,j)-^a(-i,j),  (23) 

oJ^a(i,j) ->aO',i).  (24) 

A  square  template  is  therefore  completely  symmetrical,  if  and  only  if 

a(i,  j)  =  a(-i,  j)  A  a(i,  j)  =  aO,  i)  V  (i.  j)  e  Nf(0,  0).  (25) 

The  num’oer  n|s(r)  of  independent  entries  in  a  completely  symmetrical  square  r-template  is 

n|s(r)  =  S  (i  +  1)  =  +  1.  (26) 

i  =  o  ^ 


Fig.  2;  Possible  symmetry  axes  of  square  templates  (left)  and  hexagonal  (right)  templates. 


Similarly,  Fig.  2  also  shows  the  symmetry  axes  of  hexagonal  templates.  In  hexagonal  coordi¬ 
nates,  the  actions  of  and  are  defined  by 

4°^a(i,j)-^a(-i,i+j),  (27) 

a(i,j) ->a(j,  i),  (28) 

and  consequently  a  hexagonal  template  is  completely  symmetrical  if  and  only  if 

a(i,j)  =  a(-i,i+j)  A  a(i,j)  =  aO,i)  V(i,j)€  N{‘(0,  0).  (29) 

As  the  odd  numbered  axes  intersect  cells  at  even  distances  from  the  center,  but  run  between 
pairs  of  cells  at  odd  distances,  one  best  constructs  different  formulas  for  the  number  n^5(r)  of 
independent  entries  in  completely  symmetrical  hexagonal  r-templates  for  odd  and  even  n 

n5s(r)|oddr  =  2  X  i  =  L^(L^+l)  =  i(r2  +  4r  +  3)  =  ^^  +  r+l,  (30) 
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(31) 


n5s(r)|cven  r  =  n&s(r  +  1)  -  (|  +  l)  =  j  +  r  +  1. 


With  the  GauB  bracket  []:  R  -» Z,  which  yields  the  largest  integer  smaller  than  or  equal  to  its 
argument,  one  observes  that,  for  odd  r. 


ril  1 

■  4.oddr  4 


(32) 


which  allows  us  to  merge  (30)  and  (31)  into  a  single  final  formula  for  n5s(r): 

n&s(r)  =  [^]  +  r+l.  (33) 

For  large  r,  completely  symmetrical  hexagonal  r-templates  contain  only  about  half  as  many 
independent  entries  as  the  corresponding  square  ones.  Also,  the  total  number  of  independent 
elements  is  much  smaller  than  in  the  isotropical  case. 


4)  Conclusion 

For  reference  purposes,  the  results  on  the  total  numbers  of  independent  entries  in  unrestricted, 
isotropical  and  completely  symmetrical  square  and  hexagonal  templates  are  compiled  in  Table  1. 


square 

hexagonal 

unrestricted 

4r2  4r  +  1 

3r2  +  3r  +  1 

isotropical 

r^  +  r  -1- 1 

2 

completely 

symmetrical 

^.1 

1 

Table  1 :  The  number  of  independent  elements  in  symmetric  r-templates. 


The  following  main  conclusions  can  be  drawn: 

•  Symmetry  in  a  CNN  design  problem  must  be  considered  as  early  as  possible,  as 
its  proper  exploitation  greatly  reduce  the  templates'  combinatorial  complexity. 

•  Isotropical  image  processing  tasks  are  best  solved  on  hexagonal  grids,  as  in  this 
case  the  savings  due  to  symmetiy  are  much  larger  than  on  a  square  grid. 
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Abstract 

Reaction-Diffusion  equations  used  to  model  biological  patterning  and  structure  contain 
useful  principles  of  processing  in  space;  local  nonlinear  computation  plus  information  spread 
by  diffusion.  Some  years  ago  we  introduced  the  use  of  Reaction-Diffusion  Equations  as  a 
computational  paradigm  in  image  enhancement  and  analysis,  in  particular  texture-defect 
analysis.  In  this  paper  we  desire  to  bring  the  Reaction-Diffusion  processing  paradigm  to  the 
attention  of  the  CNN-community.  We  discuss  two  classes  of  processing:  First,  non-cqulibrium 
systems  which  form  periodic  ‘Turing*  structures  allowing  analysis  of  textile  images.  Second, 
systems  whicli  ars  able  to  smooth  and  segment  grey-level  images. 


1  Introduction 

Reaction-Diffusion  (RD)  systems  ate  nonlinear  vector  partial  differential  equations,  e.g.  the  two 
component  system 


2**  = /(2i,M)  +  £V*«  (1) 

where  u  is  the  vector  of  state  variables,  D  is  the  diffusion  matrix  and  /r  represents  parameters  in 

the  nonlinear  vector  function  /(.)  In  a  famous  paper  written  to  explain  the  existence  of  pattern  and 
structure  in  living  systems,  Alan  Turing  showed  that  such  systems  could  spontaneously  generate 
patterned  structures  from  a  stable  homogeneous  initial  state,  and  that  diffusion  effects  produced 
the  instability  [9].  RD-systems  are  currently  proposed  to  explain  patterns  from  animal  coats  [3], 
shells  [2]  to  physical  systems  [8]. 

Concerning  our  image-processing  application,  let’s  take  the  example  of  textile  defect  analysis. 
The  basic  pattern-forming  property  of  RD  systems  involves  initial  growth  of  a  large  number  of 
basis  pattern  modes,  e.g.,  fourier  plane-waves,  followed  by  selection  of  a  few  interacting  modes 
by  the  nonlinearity  to  form  a  stable  pattern.  For  a  given  parameter  set,  several  solution  patterns 
may  exist.  If  any  of  these  patterns  is  similar  to  an  applied  noisy  textile  image  with  a  defect,  the 
system  will  rapidly  move  to  the  closest  pattern  in  all  regions  of  the  image  except  where  the  defect 
is.  Here  the  solution  will  be  a  different  pattern  with  different  numerical  values,  and  leads  to  easy 
detection  of  the  defect.  This  approach  was  detailed  in  [4]  [5]. 


2  Periodic  Structure  Formation 

Here  we  shall  present  the  key  stages  in  the  analysis  of  one  particular  system  emphasising  details  of 
how  it  performs  as  described  above.  We  take  the  a  variant  of  the  Fitzhugh-Nagumo  equation  (see 
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[3]),  originally  used  to  model  nerve  pulse  propagation,  which  our  analysis  shows  is  the  simplest 
useful  nontrivial  RD  system; 


^  =  /(Ui)  -M2  + 

—  =  j  («i  -  tt2  +  V^ui)  (2) 

where  the  nonlinear  function  is  taken  as  /(u)  =  au  +  the  parameters  and  y  allowing 

us  control  over  the  mix  of  quadratic  and  cubic  nonlinearity.  We  think  of  (2)  as  having  the  following 
general  form 


u  =  ^u  +  £u  +  ^(u)  (3) 

where  ^(0)  is  the  lineuization  of  the  functions  in  (2)  evaluated  at  the  operating  point  u  =  (0, 0), 
(to  avoid  ‘excitable  behaviour’),  and  ^is  a  nonlinear  operator.  Consider  first  the  linear  problem. 
If  we  had  been  working  in  continuous  physical  space  with  periodic  boundary  conditions,  we  should 
look  for  a  solution  as  an  expansion  in  plane  waves  with  wavevectors  kj .  The  linear  operator  would 
become  ^  =  i(0)  —  kjQ  and  out  expansion  would  be  in  terms  of  the  vector  eigenfunctions  of  L, 

-  =  5  H  >  +  cc  (4) 

We  use  the  bra-ket  notation  for  clarity,  since  in  general  ^  is  not  self-adjoint  so  we  need  the  left 
eigenvectors  too.  Bra’s  and  ket’s  ate  parametrized  by  n,  the  upper  (u)  or  lower  (/)  eigenpair  of  L, 
a  the  linear  part  of  the  function  in  (1)  which  we  shall  take  as  a  bifurcation  parameter,  and  the 
spatial  frequency  k.  Equation  (4)  says  the  solution  is  a  sum  over  plane  waves  each  of  which  is  a 
combination  of  two  vectors  giving  the  contributions  of  components  «i  and  uj  to  j*.  Working  in 
the  discrete  physical  space  of  our  CNN,  the  above  expansion  holds,  only  the  linear  operator  must 
be  modified, 


where  g  is  the  fourier  transform  of  the  discrete  diffusion  operator.  Taking  this  as  coupling 
to  4  neatest  neighbours,  (see  [6]  for  details  of  the  8-neighbour  operator),  the  linear  problem 
^|nak  >  =  <rna(k)\nak  >  then  defines  an  upper  and  a  lower  eigenvalue  as  a  function  of  spatial 
frequency  k.  These  ditpersion  curvet  ate  shown  in  Fig.l(a).  A  band  of  unstable  modes  is  seen 
which  will  define  the  evolving  pattern.  A  full  analysis  of  the  linear  problem  [7]  allows  us  to  choose 
D,  e  and  a  to  satisfy  various  requirements  additional  to  this  unstable  band;  no  time-periodic 
solutions,  no  chaotic  solutions  and  the  band  peak  at  a  user-definable  frequency. 

Now  let’s  outline  the  technique  of  nonlinear  analysb  which  proceeds  by  substituting  (4)  into 
(3),  multiplying  by  <  noj|e“’^  -  and  integrating  over  all  space.  This  yields  a  series  of  equations 
for  the  time-varying  amplitudes,  eg, 

iuj  =  O'uaj^uj  +  uajl^jmak  >lnal  >  -f  ...cubic  terms  (6) 

^  mn  ki 

where  Ni  =  <  uaJluaj  >  and 

<  uajl^lmafe  >|na/  >  =  (<  >),(|nof  >),.  (7) 

J>JT  «  ^ 

Here  the  subscripts  on  the  bra’s  and  kets  refer  to  their  Cj’th  component.  Remarkably,  for  our 
system  (2),  the  quadratic  and  cubic  nonlinear  interaction  coefficients  evaluate  to  ^  and  y  respec¬ 
tively.  We  simplify  the  amplitude  equation  set  by  simple  truncation:  All  lower  modes  are  ignored 
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since  their  amplitudes  are  very  small  due  to  aiaj  being  large  and  negative.  Also  since  our  pattern 
will  be  produced  by  the  frequency  at  the  centre  of  the  band,  we  see  that  harmonic  modes  can 
be  neglected  for  the  same  reason.  Fig. 1(a)  shows  that  the  dc-mode  cannot  be  neglected,  so  the 
lowest-order  system  of  amplitude  equations  describing  the  formation  of  a  pattern  in  2D  becomes 

f j  =  +  0  ^2^1  +  0  iiil  —  7  —  A'2 

ii  =  +  0  iiit  +  0  tiii  —  7  -  Aj 

U  =  ^r4«4+/3'6f2*+^'^4^i-7'f666*-A4  (8) 

where  is  the  dc-mode  and  ^21^81^4  have  equal  |fc|  shown  in  Fig.2(b)  and  where  /3*  =  P/N\  etc. 
The  terms  in  (8)  have  been  selected  by  the  above  analytical  procedure  since  the  wavevectors  in 
the  corresponding  interaction  sum  to  zero,  e.g.,  the  term  ^2(4  in  the  equation  for  (3  is  selected 
since  — ^  +  ^2+^4  =0-  Shown  in  Fig. 2  this  corresponds  to  a  simple  blob  pattern.  In  (8)  the  Xj 
represent  the  sum  of  self  and  cross  damping  terms  found  in  each  mode  equation,  with  the  form 
$|/4  and  3^j  |4|j/4,  except  for  These  terms  always  act  to  bound  the  growth  of  all  modes, 
while  those  terms  explicitly  shown  in  (8)  cause  all  modes  to  grow.  Resonances  can  be  seen,  eg 
mediated  by  the  quadratic  nonlinearity  between  modes  {2>(3>{4-  A  detailed  discussion  of  sets  of 
amplitude  equations  and  their  properties  can  be  found  in  [7].  Setting  (2  =  ^3  =  (4  &nd  assuming 
that  is  small,  we  can  solve  ihe  equation  o'(a)  +  P  i  =  Oasa  function  of  a  and  obtain 

the  solution  curve  for  the  blob  pattern,  Fig.l(b).  We  have  also  obtained  a  solution  diagram  of  a 
10-mode  truncated  system  using  continuation  software.  Blobs  bifurcate  transcritically. 


Figure  1;  (a)  Dispersion  curve  showing  upper  and  lower  <r’s,  <t„  showing  a  band  of  unstable 
modes.(b)  Solution  diagram  from  (8);  full  lines  show  stable  solution  and  dotted  unstable. 

Let’s  now  look  how  we  can  use  this  system  to  find  a  defect  in  a  textile  image  as  shown  in 
Fig. 3.  We  assume  that  we  find  a  range  of  parameters  where  the  RD-CNN  system  supports  the 
particular  textile  pattern  and  spatial  frequency,  here  we  continue  with  a  hexagonal  blob  pattern. 
If  we  apply  a  textile  image  with  defect,  Fig.3(a),  to  the  RD-CNN  system,  then  the  modes  in  (8) 
and  interactions  between  them  will  be  activated,  leading  to  growth  of  the  blob  pattern,  at  least  in 
area:  of  correct  textile.  In  the  defect  region,  growth  to  blob  pattern  will  occur  more  slowly  if  at 
all,  since  an  incorrect  mix  of  spatial  frequencies  is  present  (usually  too  many  leading  to  excessive 
damping).  We  have  differential  growth  rates  between  defect  and  ok  regions  so  that  when  the  ok 
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Figure  2:  Blob  pattern  (a)  made  of  three  plane  waves  and  (b)  their  associated  wavevectors  which 
sum  to  zero. 


region  has  reached  its  ’stable  aymptotic  state’  (SAS  ),  the  defect  regions,  if  still  evolving,  have  a 
low  numerical  value.  Defect  detection  proceeds  by  simple  thresholding  at  the  OK  SAS  numerical 
value  determined  previously. 

The  exact  details  of  this  differential  growth  (described  in  [7])  depend  on  several  factors,  one 
of  which  is  correlation  length  effects:  A  wide  band  of  unstable  modes  implies  a  short  correlation 
length  in  space,  so  that  defect  and  OK  areas  grow  more-or-less  independently.  If  the  RD-CNN 
system  supports  several  patterns  for  the  parameters  used,  then  each  region  may  evolve  to  an 
independent  pattern  each  with  its  own  SAS  .  In  our  example  here,  there  are  two  patterns  supported, 
blobs  and  rolls.  Figure  3(c)  sho^^s  localized  evolution  to  these  independent  patterns  for  a  short 
correlation  length  system.  Conversely,  when  the  unstable-mode  bandwidth  is  made  narrow,  then 
the  correlation  length  is  large,  and  the  differential  growth  of  a  single  pattern  described  above 
occurs.  The  concept  of  a  parametrically  tuned  effective  correlation  length  in  a  CNN  is  interesting. 


Figure  3:  Texture  Defect  Analysis,  (a)  Original  image  (b)  SAS  (c)  Thresholded  at  pure  blob  SAS 
level.  The  defect  is  detected  as  an  absence  of  the  correct  sized  blob.  Parameters  :  D  =  0.5618,  a  = 
0.95,  /3  =  0.2828, 7  =  1.5,  c  =  0.25 


3  Image  Smoothing  and  Segmentation 

Computer  vision  requires  a  segmentation  of  raw  image  input  into  object  and  background.  Classical 
approaches  either  maximize  regions  of  similar  image  information,  using  principles  of  continuity  and 
coherence,  or  else  look  for  edges,  using  ideas  of  dissimilarity  and  compactness.  Operators  which 
are  able  to  smooth  noise  out  of  homogeneous  regions  while  preserving  discontinuities  are  in  great 
demand,  as  are  detectors  of  lines,  curves  and  other  geometrical  shapes.  Out  early  investigations 
of  RD  systems  looked  at  their  ability  to  produce  non-periodic  localized  solutions,  isolated  lines 
and  blobs,  as  would  be  found  in  a  typical  image  of  blood  cells,  satellite  photos,  etc.  We  found 
for  several  systems  a  useful  ability  to  respond  selectively  to  objects  of  certain  scale.  There  was 
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even  evidence  of  grouping  of  object  parts  together  in  a  hierarchical  manner.  Unfortunately  these 
systems  have  not  yet  yielded  to  analysis,  and  remain  rather  ad  ujc.  We  nevertheless  feel  they 
deserve  more  attention.  Our  general  approach  was  to  invoke  several  fields,  coupled  nonlinearly. 
Each  field,  a  component  in  a  RD-CNN  vector,  represented  an  image  property.  Our  knowledge 
about  the  real  world  and  imaging  process  could  be  incorporated  in  the  (nonlinear)  relations  hard¬ 
wired  into  the  equations.  Let’s  take  a  simple  example.  Consider  two  fields,  /(*)  the  image  field 
which  is  to  be  derived  from  the  raw  data  0{x),  and  a  discontinuity  field  ff(a).  We  have  suggested 
the  following  equation  set 


a/ 

dt 
dg 
^  di 


+7iV/|-l-Z?gV^g 


(9) 


First  the  discontinuity  field  d(2.)i  produced  by  image  gradient  information  |V7|,  and  decays 

naturally  with  time  constant  Hg  and  diffuses  out  with  diffusion  constant  Dg  which  is  usually 
small.  We  find  that  g  accumulates  close  to  discontinuities  in  the  image,  giving  an  edge  map. 
Now  the  image  field  !{$);  this  is  produced  from  the  raw  input  data  0{x)  by  the  first  term,  and 
decays  naturally  with  time  constant  k/,  which  maintains  stability.  The  evolving  image  data  I 
diffuses  to  smooth  over  homogeneous  regions,  but  not  over  edges,  where  the  large  discontinuity 
field  g  effectively  reduces  the  diffusion  coefficient.  Typical  results  on  a  satellite  image  are  shown  in 
Fig. 4;  diffusion  is  hardly  apparent  over  the  strong  discontinuties.  Other  equation  sets  have  been 
developed  including  line,  junction  and  cornet  fields,  but  at  the  moment  we  know  of  no  general 
mathematical  design  approach  for  these  systems. 


a 

§; 

■B 

Figure  4;  Original  satellite  agricultural  image  (a)  segmented  and  smoothed  in  (b)  while  preserving 
d’scontinuities. 


4  Summary 

Reaction- Diffusion  systems  can  be  designed  to  have  useful  cooperative  properties  enabling  them  to 
analyze  and  enhance  grey- value  images.  In  the  case  of  periodic  structures,  analytical  and  numerical 
techniques  are  advanced  enough  to  allow  engineering  of  systems.  We  hope  similar  techniques  may 
soon  be  found  to  allow  us  to  replace  the  ad  hoc  designs  we  use  at  the  moment  with  more  rigorous 
systems. 
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Abstract 

In  the  last  years  completely  integrable  Hamiltonian  systems  were  of  great  interest  because  of 
their  physical  nature,  e.g.  the  existence  of  soliton  solutions,  and  their  relation  to  eigenvalue  and 
sorting  problems.  But  until  recently,  they  fourd  little  interest  among  electrical  engineers. 

Under  certain  restrictions,  cellular  neural  networks  (CNN)  come  very  close  to  some  Hamil¬ 
tonian  systems,  therefore  they  are  potentially  useful  for  simulating  or  realizing  such  systems.  In 
this  paper,  we  will  show  how  to  map  two  one-dimensional  nonlinear  lattices,  the  Fermi-Pasta- 
Ulam  lattice  and  the  Toda  lattice,  onto  a  CNN.  We  demonstrate  for  the  Toda  lattice,  what 
happens,  if  the  signals  are  driven  beyond  the  linear  region  of  the  PWL  output  function.  Though 
the  system  is  no  longer  Hamiltonian,  numerical  experiments  reveal  the  existence  of  solitons  for 
special  initial  conditions.  This  interesting  phenomenon  is  due  to  a  special  symmetry  in  the  CNN 
system  of  ODE’s. 

1  Introduction 

In  recent  years,  completely  integrable  Hamiltonian  systems  have  attracted  great  interest.  Closely 
related  to  them  is  the  existence  of  soliton  solutions.  The  Fermi-Pasta-Ulam  lattice  and  the  Toda 
lattice  are  classic  examples  of  such  systems  [1]  [2].  The  Hamiltonian  structure  of  such  systems 
implies,  among  other  things,  the  physical  property  of  losslessness  with  respect  to  energy.  Due  to 
the  discovery  of  systems,  like  the  Toda  lattice,  wUch  ue  capable  of  solving  eigenvalue  problems 
[3],  a  lot  of  important  applications  have  become  possible.  Consequently,  any  task  which  can  be 
formulated  as  an  eigenvalue  problem  is  a  potential  application,  e.g.  rank  filtering  [4],  [5],  [6]. 

A  lot  of  current  interest  in  electrical  enpneering  has  been  focussed  on  regular  architectures  of 
analog  nonlinear  processing  cells  with  local  connections,  called  cellular  neural  networks  [7],  [8]. 
In  general,  they  are  not  lossless  and  so  soliton  solutions  do  not  seem  to  exist.  Recently,  Roska 
et.  al.  generalized  the  original  CNN  structure  to  allow  fairly  general  nonlinesu:  and  delay-type 
feedback  and  control  [9]. 

In  this  paper  we  demonstrate  two  things.  First  we  show  under  what  conditions  the  Fermi- 
Pasta-Ulam  (FPU)  lattice  can  be  realized  exactly  by  a  conventional  CNN.  Afterwards  we  map 
a  transformed  version  of  the  Toda  lattice  [10],  [11],  [3]  onto  the  current  framework  of  CNN. 

The  original  CNN  equations  contain  a  piecewise-linear  (PWL)  map  of  the  states  with  a 
saturation  characteristic.  This  nonlinearity  can  destroy  the  integrability  of  the  Toda  lattice 
equations.  But  surprisingly  under  certain  conditions  it  is  stiU  possible  to  find  soliton  solutions. 
Tne  ’’invariants”  of  motion  are  no  longer  invariant,  i.e.  the  system  is  not  lossless  but  they  exhibit 
an  interesting  recurrent  phenomenon.  These  recurrent  phenomenon  stiU  allows  sorting,  subject 
to  some  additional  interpretations. 

2  Cellular  Neural  Networks 

The  CNN  is  a  nonlinear  dynamical  system  of  autonomous  ordinary  differential  equations  (ODE) 
[7],  [8].  If  the  state  variables  v*iy(t)  are  arranged  in  an  Af  x  IV  rectangle,  then  only  the  states 
in  a  local  neighbourhood  of  any  particular  state  v„j  are  coupled  to  the  state  v„j .  Using  the 
current  definition  of  CNN  nonlinear  templates  A  and  B  which  are  functions  of  internal  states 
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cind  input  signals,  as  well  as  time  delays  are  allowed.  In  this  paper,  however,  only  delay  free 
polynomial  type  nonlineaxities  are  admitted.  The  equations  describing  the  most  recent  class  of 
CNNs  without  time  delay  with  1  <  i  <  M\ I  <  j  <  N  are  as  follows: 

State  Equations: 

A{i,j-,k,l)vyi,i{i)+  Y,  B{i,j-,k,l)v^ki{t)  + 

^  C(*,i)e^rij  C{k,l)&Nri,} 


Output  Equations: 


\  (l'yx,i(f)  +  ij  -  - 1|) , 

(2) 

Input  Equations: 

—  Fij, 

(3) 

Constraint  Equations: 

<  1,  ^  1, 

(4) 

Parameter  Assumptions: 

C  >  0,  Rx  >  0. 

(5) 

For  the  purpose  of  this  paper  we  will  relax  some  of  the  assumptions  to  tailor  to  our  needs, 
become  clear  in  the  context  of  the  lattice  mapping. 

They 

3  Lattice  Equations 

In  general,  one- dimensional  lattices  are  described  by  a  set  n  of  structurally  identical  ODEs  (first 
or  second  order)  with  local  coupling  between  the  states  ijt  =  /(®*_i,a:jk,®*+i) .  A  special  case 
of  these  lattices  are  one-dimensional  mechanical  system  consisting  of  n  mass  points  connected 
by  nonlinear  springs,  i.e.  a  system  of  coupled  oscillators.  It  is  governed  by  the  equations  of 
motion; 

ifc  = /’(ifc  -  -  ifc)  A;  =  1,2, ...,h.  (6) 

with  boundary  conditions  at  both  ends  and  initial  conditions  for  all  mass  points. 

Quadratic,  cubic  or  piece- wise  linear  spring  characteristics  were  investigated  in  the  fourties 
and  early  fifties  by  Fermi,  Pasta  and  Ulam  [1].  They  observed  that  a  sinusoidal  initial  condition 
for  ifc  is  repeated  periodically,  though  the  system  is  nonlinear.  Related  to  this  recurrence 
phenomenon  is  the  existence  of  soliton  solutions  of  the  lattice  equations. 

Toda  investigated  the  discrete  nne-dimensional  lattice  with  the  nonlinear  spring  characterized 
by  cin  exponential  force  vs.  r  relationship,  where  r  denotes  the  distance  between  the  mass  points 
[2],  F{t)  =  exp(-r)  —  1,  and  where  the  outer  mass  points  are  fixed  at  iq  — oo  and  Xn-n  —*■  oo. 
It  can  be  shown,  both  iinalytically  and  experimentally,  that  solitons  exist  [13].  The  equations 
of  Toda  were  the  origin  of  a  series  of  mathematical  developments  after  Symes’  discovery  of 
an  intimate  relationship  between  the  Toda  lattice  and  the  eigenvalue  problems  of  symmetric 
tridiagonal  matrices  [3].  With  a  nonlinear  coordinate  transform  [10]  the  equations  of  motion  (6) 
with  exponential  force  can  be  written  as 

i,  =  2(6j-6L,)  (7) 

h  =  h  (ajfc+i  -  Ofc) .  (8) 

In  the  following  we  will  call  these  equations  the  Toda  lattice  equations,  since  they  axe  homeo- 
morphic  to  the  Toda  lattice  .  If  the  initial  conditions  for  a^,  bk  are  specified  as  the  entries  of  a 
symmetric  tridiagonal  matrix  A(0)  with  diagonal  entries  a,  and  off-diagonal  entries  6^,  then  the 
fijced  point  of  (7),  (8)  is  characterized  by  ajt  =  A*,  6*  =  0  where  A*  are  the  eigenvalues  of  A(0) 
with  the  additional  property  that  Aj  >  A2  >  ...  >  A„,  i.e.  they  come  out  sorted.  As  is  well 
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Figure  1:  Mapping  of  the  Toda  lattice  equations  onto  a  la  CNN  architecture.  The  variables  in 
the  cells  denote  the  equation  that  is  realized  by  the  respective  cell 


known,  a  property  of  completely  integrabie  Hamiltonian  systems  is  the  existence  of  as  many 
invariants  of  motion  Hk  as  the  degrees  of  freedom.  The  invariant  H\  denotes  the  trace  of  the 
matrix  A(0)  and  H2  denotes  the  Frobenius  norm  of  A(0).  The  system  posesses  n!  fixed  points 
according  to  the  n!  permuations  of  A*. 


4  CNNs  and  Lattice  Equations 

After  presenting  the  equations  for  CNN  with  unspecified  matrices  A,  B,  A,  B  and  the  equations 
for  the  FPU  and  the  Toda  lattice,  we  are  now  ready  to  relate  both  systems.  In  the  case  of  a  first 
order  ODE  lattice  equation,  the  states  of  the  lattice  equation  are  identical  to  the  CNN  states. 
The  lattice  equations  from  mass  spring  modds  are  second-order  ODEs  in  time  with  one  spatial 
dimension,  while  the  CNN  equations  are  first-order  ODEs  with  two  spatial  dimensions.  Hence, 
we  have  the  choice  to  use  either  a  double-layer  CNN  or  a  2  X  n  cell  single-layer  CNN.  A  special 
choice  for  the  template  matrices  A  and  A  make  the  structure  absolutely  regular  (Fig.  1  for 
Toda  lattice).  This  works  well  under  the  condition  that  two  rows  and  columns  (shaded  area)  of 
cells  deliver  zero  signals  to  the  cells. 


4.1  Fermi-Pasta-Ulam  Lattice 

The  FPU  lattice  with  the  PWL  spring  characteristic  fits  perfectly  into  the  original  CNN  structure 
with  the  coordinate  transformation  Vxij  =  —  Xi_i  and  the  template 


(9) 


where  all  other  templates  are  zero.  The  scaling  factor  K  has  to  be  adjusted  such  that  Vzij  does 
not  leave  the  linear  region  of  the  output  function.  Only  the  states  Vxij  are  forced  to  the  linear 
region  of  the  output  map.  In  this  way  simple  experiments  of  the  FPU  lattice  can  be  performed 
on  this  CNN.  The  mapping  of  the  FPU  lattice  onto  the  CNN  is  exact. 


4.2  Toda  Lattice 


The  Toda  lattice  equations  can  be  further  simplified  by  a  transformation  c/,  =  b\,  proposed  in 
[6],  and  a  linear  time  scaling  to  drop  the  factor  2. 

In  detail  the  following  template  matrices  are  to  be  implemented: 


■  0  0  O' 

r  0  /ij  /i3 

A  = 

0  1/Rx  0 

.  A  = 

00  0 

_  0  1  -1 

[00  0  . 

/la  = 

-VyijVykl, 

/l3  = 

WjijVytl. 

(10) 


We  call  this  system  the  modified  Toda  lattice.  The  term  1/.^  is  useful  only  in  the  presence  of 
Rx  <  00.  It  makes  the  system  less  dissipative,  but  the  choice  of  Rx  is  very  restrictive  {Rx  >  Rx) 
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for  stability  reas-.rL:'.  ils  opposed  to  the  FPU  lattice,  the  mapping  of  the  Toda  lattice  is  exact 
if  all  states  utilize  only  the  linear  part  of  the  output  function.  Bounds  for  the  initial  values 
can  be  given. 


4.3  Modified  Toda  Lattice 


Because  of  the  nonlinear  output  map,  the  CNN  does  not  exactly  behave  like  a  Toda  lattice 
for  initial  conditions,  which  not  meet  the  bounds.  The  saturation  characteristics  bounds  the 

<  2.  This  implies  bounded  acceleration  and  propagation  speed 
ue  to  the  boundary  conditions  bo  =  bn  =  0,  the  fixed  point  is  still 
given  by  bk  —  0.  This  follows  by  induction  from  the  boundary  conditions  and  the  equations  for 
hk-  As  for  the  Toda  lattice,  no  information  on  the  a*  is  available. 


derivatives,  namely  jajt 
in  terms  of  the  Toda 


,  WiliCli  uc 

i;ci  <  2,  |6* 
lattice.  Di 


5  Numerical  Experiments  with  Modified  Toda  Equations 

This  section  summarizes  numerical  experiments  on  the  modified  Toda  lattice  ((1)  -  (5)  and  (10)) 
with  a  PWL  output  map  (2).  The  FPU  lattice  is  not  considered  here  because  we  only  wanted 
to  show  that  a  CNN  could  be  used  for  an  electrical  circuit  realization  of  this  lattice. 


5.1  The  Most  Simple  System 

At  first  3  examples  for  a  2  X  2  matrix  axe  investigated. 
Example: 


A(0)  = 


-2.0 

0.1 


0.1 

0 


The  transients  for  ai,  C2,  are  shown  in  Fig.  2  a,  b.  The  bounds  of  the  derivatives  stretch  the 


Figure  2;  Example,  a),  b)  trajectories  ai  ( — ),  a2  (-  -  -),  ( . ),  c)  invariants  Hi  ( — ),  H2 

( . ),  d),  e)  eigenvalues  of  the  Jacobian  matrix 


transients,  but  the  initial  values  ai(0),  02(0)  are  still  sorted  by  value.  The  system  is  no  longer 


lossless  but  can  be  lossless,  passive  or  active.  The  passive  and  active  mode  keep  a  balance  such 
that  the  invariant  H2  is  recovered  after  sorting.  The  invariant  Hi  does  not  chajige  its  value. 

The  solution  of  the  Toda  lattice  and  the  modified  Toda  lattice  differ  by  a  time  scaling  of 
■v/2.  The  invariant  H2  yields  H2  =  o?  —  5:^/2  cosh^(dt  +  7).  Hence,  H2  is  not  constant,  but 
decreasing  for  tm  <  system  is  passive,  and  increasing  for  tm  >  i-®- 

system  is  active.  So  we  have  a  symmetry  with  respect  to  tm- 

In  the  modified  Toda  lattice  for  dimension  n  =  2  with  special  initial  conditions  such  that 
the  ODE  is  still  nonlinear,  a  recurrence  phenomena  can  be  observed.  This  is  due  to  a  symmetry 
in  time,  e.g.  of  the  eigenvalues  of  the  Jacobian  matrix. 

5.2  Sorting  of  Numbers 

The  modified  Toda  equations  behave  like  a  sorter,  but  a  strict  sorting  order  no  longer  exists. 
This  is  due  to  the  limited  propagation  speed  of  the  data  and  the  stability  properties  of  the 
fixed  points.  The  Linearization  of  the  modified  Toda  lattice  about  a  fixed  point  gives:  a*  =  0, 
=  (^(ajk+i)  —  p(a*))  p'(0)6fc.  The  variables  a*  denote  the  values  of  the  diagonal  entries  at 
the  fixed  point  and  p*(0)  denotes  the  derivative  of  the  output  function.  A  fixed  point  is  called 
asymptotically  stable^  with  respect  to  perturbations  in  bk,  if  for  all  first  sub-  and  superdiagonal 
entries,  the  following  inequality  holds  (^'(O)  >  0):  g(ak+i)  —  gia.k)  <  0. 

The  inequality  gives  no  information  on  the  order  for  iik+i  and  ajt  for  >  1  and  >  1 
(— 1  equivalently).  So  their  order  at  the  fixed  point  is  undecided. 

Suppose  Idfcl  <  1  for  n,  values,  d*  >  1  for  n+  values  and  d*  <  —1  for  n_  values.  Then,  for 
the  moified  Toda  lattice  the  number  of  asymptotically  stable  fixed  points  rifp  is  determined  by 
the  number  of  permutations  of  the  elements  with  d^  >  1  and  with  dj^  <  —1,  thus  rifp  =  n+!n_!. 

If  the  output  map  is  applied  to  the  sorting  result,  a  half  order  is  realized,  since  all  values 
d*  >  1  (dfc  <  1)  reduce  to  d*  =  1  (d*  =  -1). 

The  multiplicity  in  the  fixed  points  is  caused  by  the  bounded  derivatives.  Therefore  the 
propagation  speed  for  data  |d.^|  >  1  is  equal.  So  their  final  position  depends  on  the  initial 
position. 

5.3  Soliton  Solutions 

An  interesting  result  of  the  modified  Toda  lattice  equations  is  the  existence  of  soliton  solutions. 
Such  a  solution  requires  the  initial  conditions  to  be  such  that  the  ODEs  operate  in  a  nonlinear 
regime.  Then  a  wave  with  constant  shape  is  built  up  and  two  waves  pass  each  other  without 
changing  their  shape  after  the  collision. 

In  Fig.  3  the  evolution  of  (states  vs.  time)  of  the  variables  ak  for  a  lattice  of  dimension  7 
with  initial  values 


1  •  •  •  >  ^  1 1  ^ 

■  -0.75 

0 

0 

0 

0 

0 

1.4  ] 

61,  ...,  bn-1 

0.01 

0.01 

0.01 

0.01 

0.01 

0.01 

0.01  J 

is  shown.  The  little  irregularity  in  the  shape  of  the  curves  is  due  to  a  downsampling  of  the 
simulation  result.  These  initial  values  exclude  either  a  pure  Toda  lattice  or  a  purely  linear  ODE. 
This  picture  reveals  the  typical  behaviour  of  soliton  solutions:  their  shape  is  constant  over  time 
and  is  not  changed  by  head-on  collisions. 


6  Conclusion 

In  this  paper  we  have  shown  how  two  famous  nonlinear  lattice  equations,  the  FPU  lattice  and 
the  Toda  lattice  can  be  mapped  onto  a  CNN.  For  this  purpose  certain  original  restrictions  on 
the  templates,  such  as  symmetry,  were  dropped.  Nevertheless  the  system  is  still  stable,  because 
the  underlying  mechanical  models  are  stable.  The  mapping  is  exact  under  certain  conditions  for 
the  state  variables  or  initial  conditions,  respectively.  Otherwise,  the  nonlinear  output  map  of 

^Strictly  speaking  they  are  only  stable. 
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Figure  3:  Soliton  solution  in  modified  Toda  lattice  with  head  on  collision  of  two  solitons 


the  CNN  modifies  the  lattice  equations.  This  was  demonstrated  for  the  Toda  lattice.  Numerical 
experiments  and  some  analytical  reasonings  show  that  the  sorting  operations  can  be  performed 
for  certain  initial  conditions.  It  has  to  be  stressed  that  the  system  is  not  Hamiltonian,  but 
possesses  symmetries  in  time. 
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ABSTRACT 

It  is  shown  that  the  game  of  life  algorithm,  which  is  equivalent  to  Turing  machine,  can  be  realized  by 
CNN.  Next,  a  multipath  CNN  algotrithm  is  shown  demonstrating  the  capabilities  of  analog/logic 
(analogic)  software. 


1.  INTRODUCTION  AND  THE  MAIN  RESULTS 

It  is  well  known  that  in  the  realm  of  logic  computing  Turing  machines  are  universal  in  a  sense 
that  any  conceivable  algorithm  (recursive  function)  can  be  realized  with  it.  In  the  field  of  analog  regular 
processing  arrays  it  has  been  shown  that  any  nonlinear  operator  with  fading  memory  can  be  realized  by 
at  most  4  layers  of  neural  networks  containing  memor^ess  nonlinearities  and  delays  [2],  Cellular  neural 
network  (CNN)  is  a  new  paradigm  [1]  for  locally  connected,  geometrically  placed,  analog.  3D  regular 
processing  arrays.  By  introducing  the  nonlinear  and  delay-type  templates  and  additional  capabilities  [3] 
we  have  an  extremely  broad  universe  of  functionalities.  Multi-layer  perceptrons  can  be  realized  also  by  a 
class  of  cellular  neural  networks  [4].  It  can  be  shown  that  all  the  three  types  of  partial  differential 
equations  can  be  approximated  by  CNNs  [3c].  The  programmable  CNNs  [3]  provide  a  new  quality:  a 
simple  way  to  solve  CNN  algorithms,  a  kind  of  analog  software. 

The  silicon  realization  of  CNN  is  convincing:  the  first  tested  CNN  array  has  a  capability  of  0.3 
TeraXPS  on  a  chip  [9]  with  2  micron  technology  (fixed  template)  and  the  first  programmable  dualy 
computing  CNN  chip  provides  the  full  programmability  [8]. 

Hence,  it  seems  that  the  programmable  CNN.  in  the  above  sense,  is  an  universal 
programmable  analog  array  computer. 

In  this  report  we  show  that 

-  the  game  of  life  can  be  realized  by  CNNs  which  means  that  any  Turing  machine  can  be  realized 
by  simple  programmable  CNNs. 

-  complex  image  processing  tasks  can  be  realized  by  programmable  dual  computing  CNN  by 
showing  three  useful  CNN  solutions  (one  shown  in  details) 

Thus,  we  can  prove  that  the  programmable  CNN  is,  theoretically,  a  universal  machine  in  both 
the  analog  and  the  logical  field.  Moreover,  for  a  broad  class  of  problems  the  practical  CNN  algorithms 
represent  a  highly  efficient  truly  parallel  solution.  In  view  of  recent  neurobiological  results  [10,5,6,15] 
and  our  first  biological  CNN  mr^els  [11]  it  seems  almost  certain  that  the  programmable  CNN  provides 
a  very  strong  modelling  paradigm  for  a  lot  of  living  systems. 

2.  THE  GAME  OF  LIFE  TEMPLATES 


2.1.  The  basic  step  template 

Being  a  no-player  game,  the  life  game  [7]  is  not  realty  a  game  in  its  traditional  meaning.  In 
principle  it  is  "played"  on  an  infinite  squared  board  (cell  array).  At  any  time  some  of  the  cells  will  be  alive 
and  the  others  dead.  After  setting  up  an  initial  table,  in  each  time  step,  the  next  state  is  defined  by  the 
following  rules  [7]: 
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BIRTH:  a  cell  that  is  dead  at  time  t  becomes  alive  at  time  t+l  only  if  exactly  3  of  its  eight 
neighbors  were  alive  at  time  t. 

DEATH:  a  cell  that  is  alive  at  time  t  and  has  less  than  2  or  more  than  3  alive  neighbors  at  time  t 
will  be  dead  by  time  t  + 1 . 

SURVIVAL:  a  cell  that  was  alive  at  time  t  will  remain  alive  at  t+ 1  if  and  only  if  it  had  exactly  2  or 
3  alive  neighbors  at  time  t 


The  facts,  that  the  cells  of  the  array  can  be  projected  onto  those  of  the  CNN  arid  the  new  state 
of  a  cell  is  determined  only  by  the  previous  state  of  it  and  that  of  its  neighbors,  just  like  with  the  CNN, 
are  appealing  to  realize  the  game  with  a  CNN.  The  2  layer  network  of  Fig.  1 .  makes  one  step  of  the 
game. 
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Fig.  1:  The  2  layer  single-step  game  of  life  template.  The  initial  state  has  to  be  fed  on  the  input 
of  layer  1,  and  the  new  state  will  appear  on  the  output  of  layer  2. 

To  understand  how  this  network  works,  let’s  reformulate  the  rules  of  the  game:  a  cell  will  be 
alive  if  at  least  3  of  the  9  cells  in  Its  3x3  neighborhood  are  alive  and  at  most  3  of  its  8  neighbors  are 
alive.  Layer  one  implements  the  first  part  of  this  conjunction,  layer  2  realizes  the  second  part  and  the 
conjunction  itself  (see  Fig.  2.). 


Fig.  2:  The  singie-step  iife  algorithm,  a:  Input  pattern,  b:  output  of  layer  1,  c:  new  state  (output 
of  layer  2) 


2.2  Versions  of  a  multistep  CNN  aigorithm 


2.2. 1  Discrete  Time  CNN  with  thresholding  sigmoid 

The  algorithm  described  in  the  previous  chapter  executes  a  single  step  of  the  life  game  with 
one  transient.  It  would  be  much  more  useful,  if  we  could  simulate  a  whole  life  history  with  one  CNN,  i.e. 
the  transient  would  settle  down  only  if  the  game  reached  a  stable  state,  otherwise  it  would  run  forever 
(see  [7]  for  some  stable  and  oscillating  patterns  and  the  famous  forever  growing  glider  gun).  To  reach 
our  goal,  two  transients  should  be  composed  somehow;  one  transient  between  two  states  and  another 
from  the  initial  state  to  a  final  state  or  to  eternity.  For  this  reason  the  output  should  be  fed  back  to  the 
input,  inducing  a  new  transient.  But  there  is  a  problem  with  timing;  a  new  step  can  begin  only  after  the 
termination  of  the  previous  one.  Unfortunately,  simply  feeding  back  layer  2  of  the  above  described  CNN 
to  the  input  does  not  meet  this  condition,  because  the  time  required  to  reach  the  new  state  (all  celts 
have  values  ±1)  depends  on  the  number  of  neighbors,  i.e.  it  alters  from  cell  to  cell,  causing  a  terrible 
confusion.  This  problem  can  be  solved  most  easily  with  a  Discrete  Time  Cellular  Neural  Network 
(OTCNN)  [12],  which  is  equivalent  to  the  CNN  calculation  using  a  fonvard  Euler  iteration  with  stepsize  1 
and  thresholding  sigmoid  [13].  Here  the  output  state  of  a  cell  is  binary  (±1)  and  is  determined  by  the 
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sign  of  the  state  of  the  cell.  The  above  described  algorithm  needs  to  be  slightly  modified:  three  layers 
are  required,  one  for  each  part  of  the  conjunctior.  and  a  third  layer  to  realize  the  conjunction  itself, 
yielding  the  output  pattern.  (Fig.  3.). 
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Fig.  3:  The  3  layer  DTCNN  template  executing  the  whole  life  history. 


2.2.2  Discrete  Time  CNN  with  piecewise-linear  thresholding 


Another  solution  can  be  achieved  with  a  DTCNN  using  piecewise-linear  thresholding,  just  like  in 
the  continuous-time  CNN.  Using  this  network  the  output  can  take  any  value  between  -1  and  +1.  These 
values  can  be  used  to  code  information  about  the  appropriate  cell.  Using  nonlinear  templates  [3a,b]  we 
can  extract  the  coded  information.  The  template  of  Fig.  4.  realizes  the  multistep  game  of  life  algorithm  in 
one  layer .  The  first  step  of  the  algorithm  is  to  code  the  information  about  the  previous  state  and  the 
number  of  alive  neighbors  of  the  current  cell,  and  the  second  step  is  to  decode  this  information,  i.e.  to 
drive  the  pixel  black  or  white,  depending  on  the  coded  value.  This  means  that  a  new  life  pattern  appears 
after  every  second  iteration  step.  The  result  of  the  first,  coding  step  is; 

base  number  +  0.1  *  number  of  alive  neighbors 

where  the  base  number  is  -0,5  and  -0.45  for  the  white  (-1)  and  black  (+1)  cells,  respectively.  Function  b 
"counts"  the  base  number  and  function  a  the  number  of  alive  neighbors  at  x=±1 . 


Consequently,  the  second  step,  the  decoding  phase  should  drive  the  pixel  black,  if  the  output 
of  the  first  step  is  in  the  [-0.25,-0.15]  domain.  This  is  where  the  value  of  function  5  is  + 1 . 
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Fig.  4:  Multistep  game  of  life  template  for  piecewise-linear  DTCNN 


3.  MULTIPATH  CNN  ALGORITHMS 

Due  to  local  interconnectivity  the  CNN  seems  to  be  mainly  suitable  for  detecting  local 
properties,  however  some  global  properties  can  be  extracted  as  well.  Moreover,  using  the  programable 
CNN  multipath  algorithms  even  more  complex  problems  can  be  solved. 

3.1  Blob-counting 

Our  task  is  to  count  blobs  in  a  grey-scale  image.  The  key  of  the  algorithm  is  to  associate  a  blob 
with  a  well-defined  significant  point  and  after  suppressing  the  other  pixels  we  only  have  to  count  these 
significant  points.  Shifting  the  remaining  pixels  to  one  side  of  the  picture  (e.g.  with  horizontal  CCD  [14]) 
the  counting  of  them  becomes  easy. 

The  basic  idea  of  the  algorithm 

Let's  associate  each  blob  with  its  local  southern  places  (i.e.  with  pixels  having  neither  south¬ 
eastern,  nor  southern,  nor  south-western  direct  neighbors).  In  two  cases  a  blob  can  have  more  than 
one  southern  place: 
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(i)  The  bloD  has  a  horizontal  southern  edge  (Fig.  5e); 

(ii)  The  blob  is  concave  from  the  bottom  (Fig.  5b.5d  and  their  vertical  mirrors); 

But  fortunately  neither  causes  any  problem,  because  in  case  (i)  the  horizontal  CCD  contracts  the 
soutriern  places  into  a  single  pixel,  and  in  case  (ii)  each  concave  place  yields  one  new  local  southern 
place,  so  the  difference  of  the  number  of  the  local  concave  places  and  the  southern  places  gives  the 
correct  result,  ie.  the  number  of  the  blobs. 
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Fig.  5:  In  3X3  neighborhood  these  bottom  edges  (and  their  vertical  mirrors)  could  occur 
(a,b,c,d,e).  The  LCP  template  extracts  two  pixels  from  location  (f).  The  I  sign  denotes 
the  blob  pixels,  and  the  •  sign  denotes  the  background  pixels. 

Holes  can  cause  problems,  because  their  local  northern  point  is  local  concave  place,  but  it  doesn’t 
increase  the  number  of  the  local  southern  places.  So  if  we  want  to  know  the  number  of  the  blobs  we 
have  to  fill  their  holes  first.  Unless  we  do  it,  the  algorithm  gives  the  Euler  number  of  the  image,  i.e.  the 
difference  of  the  number  of  blobs  and  the  number  of  holes. 

The  templates  and  the  flow  diagram  of  the  algorithm 

The  complete  algorithm  is  shown  on  Fig.  7.  The  templates  of  the  algorithm  work  on  black-and- 
white  pictures,  so  the  first  step  is  a  grey-scale  to  black-and-white  transformation  using  the  average 
template  [1].  In  the  second  step  we  have  to  fill  the  holes  with  the  hole-filler  template  [14],  We  call  the 
output  of  these  transformations  the  preprocessed  picture. 

At  this  point  the  algorithm  branches  into  two  independent  parallel  paths.  In  the  first  branch  the 
Local  Southern  Elements  (LSE)  are  extracted  by  the  LSE  template  (Fig.  6a).  Next  they  are  shifted 
horizontally  with  the  CCD  transformation.  The  last  step  of  this  branch  is  to  count  the  black  pixels.  It  is 
easy  using  a  logical  algorithm  because  they  are  only  on  one  side  of  the  picture  in  determined  order.  The 
CCD  transformation  causes  in  this  case  a  large  data  reduction  without  any  information  losses. 

In  the  second  branch  the  task  is  very  similar.  The  only  difference  is  that  instead  of  the  LSE 
template,  the  Local  Concave  Places  detector  (LCP  template  Fig.  6b)  is  used.  The  LCP  template 
associates  one  pixel  with  each  local  concave  place  except  in  the  situation  depicted  on  Fig.  5f,  where 
two  pixels  are  extracted  but  the  horizontal  CCD  transformation  melts  these  two  neighboring  pixels  into 
one. 

Finally,  the  last  step  is  the  subtracting  the  number  of  LSPs  from  the  number  of  LSEs. 
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Fig.  6:  The  LSE  (a)  and  the  LCP  (b)  templates 


An  example:  Counting  kevs 

A  grey-scale  camera  picture  was  taken  of  seven  keys  with  a  resolution  of  63x62  pixels  (Fig.  8a). 
First  it  was  transformed  to  black-and-white  (Fig.  8b),  and  then  the  holes  were  filled  (Fig.  8c).  Next, 
following  the  two  parallel  branches,  the  local  southern  elements  (Fig.  8d:  24  connected  components) 
and  the  local  concave  places  (Fig.  8e;  17  connected  components)  were  extracted.  Finally:  24-17-7:  it's 
correct! 

In  case  of  a  consen/ative  analog  VLSI  realization  it  could  take  about  20  microseconds  (it 
depends  only  on  the  size  of  the  picture). 
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3.2.  Other  muKipath  CNN  algorithms 

Two  other  multipath  CNN  algorithms  were  realized.  The  first  one  can  make  blob  size 
classification.  This  probabilistic  algorithm  can  classify  larger  blobs  than  the  template  size.  The  second 
one  can  detect  textile-pattern  errors.  It  works  on  a  simple  sleazy  weaved  textile,  and  can  detect  the  fiber 
breakings  and  the  knots. 
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